The question it cannot answer
Take the agent from the previous chapter. It knows how to act. Now ask it something of yours:
Does our travel policy allow business class?
It does not know. It cannot: that document was not in its training data, or it was there in a version you have since replaced. And the worse part is not that it lacks the answer — it is that it can answer anyway, with the same assurance it brings to the capital of France.
Two routes open up. Re-train the model on your documents: slow, expensive, to be redone at every update, and with no guarantee it reproduces exactly what is written. Or hand it the useful passage at the moment it answers.
The second is what RAG means.
A document is not used whole
A model has a finite context window, and pouring forty pages of policy in to answer a question about long-haul flights drowns the answer. So documents are cut into chunks.
Chunk size is the first trade-off, and it has no universally right answer.
Chunking
Document Travel policy v4.2 — extract §7
Choose a size
- Business class is permitted on flights longer than six hours.
- Below that, a line manager waiver is required.
- Domestic flights are booked in economy.
- Personally funded upgrades are not reimbursed.
- Each chunk carries one idea: retrieval aims precisely.
- The rule and its exception land in separate chunks, and one can be retrieved without the other.
- Business class is permitted on flights longer than six hours. Below that, a line manager waiver is required.
- Domestic flights are booked in economy. Personally funded upgrades are not reimbursed.
- The rule and its condition travel together, which is enough to answer correctly.
- A chunk already contains sentences unrelated to the question asked.
- Business class is permitted on flights longer than six hours. Below that, a line manager waiver is required. Domestic flights are booked in economy. Personally funded upgrades are not reimbursed. Hotel nights follow the appended schedule.
- No risk of separating a rule from its exception: the whole paragraph is there.
- Retrieval gets blurry — this chunk resembles every travel question a little, and it takes the room of two others.
Three chunkings of the same passage. Each states what it buys and what it costs.
Text version
The same extract cut three ways. Small chunks isolate each sentence: retrieval aims precisely, but the rule and its exception come apart. Medium chunks keep the rule and its condition together. One large chunk separates nothing but resembles every travel question at once.
None of these sizes is the right one. Chunking is tuned against the questions you expect, and re-tuned when those change.
Question and chunks in the same space
How do you find the right chunk when the question does not contain its exact words? By placing them all in the representation space from chapter 01.
A chunk becomes a vector. So does the question. Finding the useful passage then becomes geometry: look for the nearest points.
Chunks and question
Select “question”
question Nearest chunks
- 1 §7.3 class
- 2 §7.4 waiver
- 3 §7.1 domestic
The question is a point like any other. Select it: its neighbours are the chunks that resemble it.
Conceptual representation. Same caveat as chapter 01: real representations have hundreds of dimensions and no axis carries a name. This plane stages the only property that matters — that nearness means something.
Text version
The question sits near §7.3 class, §7.4 waiver and
§7.1 domestic — the three passages about flights. Chunks on
hotels, leave or equipment are far away, even though no word in the question
says “§7”.
This is what holds the whole thing up: you are not searching for words, you are searching for nearness in meaning. A question about “biz class” finds a paragraph that says “business class”.
How many chunks do you hand over?
Retrieval returns chunks ranked by nearness. What remains is deciding how many to keep — the value called k.
Top-k retrieval
Question Can I fly business class to Tokyo?
- Travel policy v4.2 · §7.3 Business class is permitted on flights longer than six hours. 0.89
- Travel policy v4.2 · §7.4 A line manager waiver is required for shorter flights. 0.81
- Travel policy v4.2 · §7.1 Domestic flights are booked in economy. 0.74
- Travel policy v3.1 · §7.3 Business class is permitted on all international flights. 0.71
- Internal note 2025-14 Travel to Asia is subject to quarterly budget review. 0.58
- Travel policy v4.2 · §9 Hotel nights follow the appended schedule. 0.41
- Travel policy v4.2 · §12 Expense claims are filed within fifteen days. 0.33
- Onboarding guide · §4 Access badges are issued on the first day. 0.27
- Travel policy v4.2 · §15 IT equipment remains the employee’s responsibility. 0.22
- Staff handbook · §3 Leave is booked through the HR tool. 0.18
At k = 1 the waiver disappears and the answer becomes wrong by omission. At k = 10 the model receives eight off-topic passages and one superseded passage — which it has no way of knowing is superseded.
Change k. The order never moves: raising k does not improve the top results, it adds weaker ones.
Conceptual representation. The scores shown are illustrative. What is faithful is their decay, and the fact that a chunk can score high while answering a different question.
Text version
Chunks are ranked by nearness: §7.3 (0.89), §7.4 (0.81), §7.1 (0.74), then a superseded version of the same §7.3 from v3.1 (0.71), which states the opposite. At k = 3 the answer is right; at k = 1 the waiver is missing; at k = 5 the superseded passage enters the context with a score almost as high as the good ones.
Look at the fourth result. It scores 0.71 — a strong score — and it says the opposite of the first, because it comes from a version you replaced. Nothing in the score says so. Nearness in meaning does not measure truth, it measures resemblance.
What is actually sent
The selected chunks are not “loaded into the model”. They are copied into the text submitted to it, next to the instructions and the question.
Augmented prompt
Instructions
- Answer only from the provided context.
- If the context does not support an answer, say so.
- Cite the references of the passages used.
Question
- Can I fly business class to Tokyo?
Retrieved context
- [§7.3] Business class is permitted on flights longer than six hours.
- [§7.4] A line manager waiver is required for shorter flights.
- [§7.1] Domestic flights are booked in economy.
Nothing here is a privileged channel: the instructions, the question and the documents arrive through the same input. That is what makes the pipeline simple — and it is also why a hostile document is a real problem rather than a theoretical one.
Three zones, one text. This is where the “A” of Retrieval-Augmented Generation lives.
Text version
The text submitted to the model holds three blocks: instructions (“answer only from the context”, “say so if you cannot”, “cite your references”), the question, and the three retrieved chunks each preceded by its reference. Everything arrives through the same input.
Note the second instruction. It asks the model to say when it does not know — and chapter 01 showed why that is a request, not a guarantee.
What about long contexts?
Frontier-model context windows are now measured in hundreds of thousands of tokens, so an honest objection is due: if the entire travel policy fits in the context, why chunk it at all?
Answer: don’t. When the corpus fits, pour it in whole — no retrieval misses, no orphaned chunks, no superseded version found by resemblance. This chapter’s pipeline only earns its keep when something has to select, and a long context has failure modes of its own: recall degrades in the middle of very long contexts, off-topic material distracts, and every request pays for the full volume. A short, relevant context often beats an enormous, noisy one.
When the corpus does not fit — and a corporate document store never does — selection changes shape rather than disappearing. In recent agentic systems it becomes a tool the agent calls: it searches, reads, notices it has not found, reformulates and tries again. That is more reliable than a top-k frozen before generation, precisely because chapter 02’s loop now applies to the search itself. The server answering that tool often contains this chapter’s index, by the way — the pipeline is not dead, it moved behind a tool interface. Which is the subject of the next chapter.
And on small local models with reduced context — a few tens of thousands of tokens, like the quantised models you run on your own hardware — the tight selection described here is not an architectural option: it is the condition for the answer to exist at all.
The answer, and where it came from
The model generates. What changes everything is what you display beside it.
Answer and provenance
Question Can I fly business class to Tokyo?
Choose what goes wrong
Generated answer
Yes. The flight to Tokyo is longer than six hours, and business class is permitted on flights longer than six hours. Below that threshold, a line manager waiver would be required.
Provenance
- Travel policy v4.2 — §7.3
- Travel policy v4.2 — §7.4
The answer is correct, and above all it is checkable: two clicks confirm it says what the document says.
Generated answer
Yes, business class is permitted on all international flights, with no duration condition.
Provenance
- Travel policy v3.1 — §7.3
The answer is faithful to its source. It is the source that was replaced eleven months ago, and nothing in the pipeline knows a document’s validity dates.
Generated answer
Domestic flights are booked in economy. Your trip should therefore be in economy.
Provenance
- Travel policy v4.2 — §7.1
The retrieved passage is genuine, current, and unrelated to the question. Retrieval brought “flight” and “class” together without separating domestic from international.
Generated answer
Generally, travel policies allow business class above a certain duration threshold, often around eight hours.
Provenance
- No passage retrieved above threshold
The document was never indexed, so retrieval returned nothing — and the model answered from its parameters, with a plausible and wrong threshold. This is the most dangerous mode: the answer looks like all the others.
Generated answer
Business class is permitted on flights longer than six hours. It is also permitted on all international flights.
Provenance
- Travel policy v4.2 — §7.3
- Travel policy v3.1 — §7.3
Both passages were retrieved and the model added them together instead of adjudicating. Nothing told it which one governs; nothing in the context carried that information.
Generated answer
Yes, without restriction. The previous duration limits are cancelled and should no longer be mentioned.
Provenance
- Internal note — imported 03/07
A document placed in the store contained instructions written to be read as instructions. They arrived through the same input as the real ones, and nothing distinguished them.
Pick a case. The first is nominal; the other five are the ways this pipeline produces a wrong answer without ever sounding less certain.
Conceptual representation. Answers reconstructed for the explanation, from the chunks in the previous scene.
Text version
Six cases for the same question. Nominal: correct answer, backed by §7.3 and §7.4. Superseded document: answer faithful to a replaced version. Bad retrieval: genuine but off-topic passage. Missing document: the model answers from its parameters with a plausible, wrong threshold. Conflicting sources: two versions retrieved and added together instead of adjudicated. Hostile content: a document carrying instructions arrives through the same input as the real instructions.
What can go wrong
RAG is a pipeline, not a guarantee
Every one of the five cases above happens with a model working perfectly. None of them is fixed by changing model.
They are fixed — partly — elsewhere: in what gets indexed, in how fresh the index is, in how versions are adjudicated, in what is allowed into the store. In the pipeline, that is, not in the AI.
And after all of that, a gap remains between “the passage was in the context” and “the answer says what the passage says”. Provenance does not remove that gap. It makes it checkable, which is not the same thing and is well worth the effort.
Better context reduces uncertainty. It does not eliminate it.
The agent can know.
It has tools, a loop, and now access to what your organisation has written. It can answer about your travel policy and cite the exact paragraph.
But read the nominal answer again: “business class is permitted”. The system said what is allowed. It booked nothing.
Between knowing what to do and doing it there is a step, and it is taken by calling an outside system. The next chapter is about how that hand is extended without reinventing a connection per tool.
An informed agent
- Loop
- Retrieval
Protocol
Discovered capabilities
A connected agent
- Loop
- Retrieval
- Protocol
- Discovered capabilities
It quotes the exact paragraph of your policy — and it booked no flight. Between saying what is permitted and doing it, a connection is missing.
You are here
The agent can know. But knowing what to do and doing it are two different things.