Three papers worth knowing: Transformer, RAG and ReAct

You do not need to know dozens of works to understand how today’s AI tools work. Three that set the direction are enough: about the architecture language models are built from, about adding external knowledge to answers, and about the loop of reasoning and acting that agents rely on. This text summarizes them according to their own abstracts and points to where you will meet each idea. How to read such works yourself is described in the guide How to read AI papers.

Transformer: attention instead of recurrence

The 2017 paper Attention Is All You Need proposes the Transformer architecture. The authors note that the dominant sequence models were based on complex recurrent or convolutional networks, and the best ones also connected the encoder and decoder through an attention mechanism. They propose a simpler network based solely on attention mechanisms, with no recurrence or convolutions. In experiments on two machine translation tasks the models turned out better in quality, easier to parallelize and needed significantly less training time. Why does it matter today? Parallelization made it possible to train very large models on huge sets of text, and this work is widely described as a starting point for many later language models (a view from the field that the abstract itself does not state). When you read about the context window or tokens, you touch consequences of this work.

RAG: external memory for a model

The 2020 paper Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks starts from the observation that large models store knowledge in their parameters, but their ability to access and manipulate it precisely is limited, and providing sources for their decisions and updating their knowledge remain problems. The authors combine parametric memory, a pre-trained model, with non-parametric memory in the form of a dense vector index of documents accessed through a trained retriever. Such a model then set the state of the art on three open-domain question answering tasks. Today this pattern is the first thing you reach for when an application has to answer from its own documents. A practical description is in the article RAG step by step.

ReAct: reasoning and acting in turn

The 2022 paper ReAct notes that models’ ability to reason (for example in the chain-of-thought style) and to act (for example planning actions) had been studied separately. The authors propose generating reasoning traces and actions in an interleaved manner. Reasoning helps to build and update a plan and to handle exceptions, while actions let the model reach external sources such as knowledge bases or environments for additional information. In question answering and fact verification the approach reduced hallucination and error propagation, and gave traces more readable to humans. This is exactly the rhythm in which the agent described in the article How a coding agent works: the tool loop works.

How to read them in order

If you have one evening, read the three abstracts in chronological order: Transformer, RAG, ReAct. Each time write one sentence about the problem and one about the idea. Then look at the most important figure or table from each work, because a diagram of the architecture or process says more than a paragraph of text. Finally try to describe in your own words how these three ideas cooperate in an application you know. If you can do that without looking at your notes, you understand the foundation on which most of today’s tools stand, and you will judge marketing promises more easily.

What follows in practice

First, these ideas compose into one picture: a model based on the Transformer architecture answers from a context to which retrieval (RAG) adds fragments of knowledge, and an agent (ReAct) decides in a loop which tool to call. Second, none of the works promises magic: each describes a concrete mechanism and its limitations, which makes it easier to judge what to expect from a tool. Third, history shows that breakthroughs are often simple ideas that scale well. If you want to deepen your knowledge, start with abstracts and ask the five questions from the guide to reading papers, and you will find related materials in the Research and papers hub.

1. 01
   TransformerThe architecture of an attention-based model.
2. 02
   ContextThe model sees text in a window.
3. 03
   RAGRetrieval adds knowledge to the context.
4. 04
   ReActThe agent alternately thinks and acts.
5. 05
   ToolsAccess to data and actions, e.g. via MCP.
6. 06
   EvaluationTests check the whole.

Last checked: 20 October 2026. The abstracts of the works are summarized; consult the originals for details.