A language model knows what was in its training data. It does not know your contracts, your files, or last quarter’s correspondence. The obvious fix — paste everything into the question — stops working almost immediately: a single service contract already exceeds what a model can usefully hold at once, and an archive of them is not even close.
So something has to stand in front of the model and decide, for each question, which handful of paragraphs it is allowed to read. That decision is what “retrieval” means, and nearly everything that makes these systems good or useless happens there — before the model has generated a single word.
First the document is cut into pieces
Retrieval can only ever hand back a piece that exists. So the first step is cutting documents into chunks — usually a few sentences each — and the cuts are not a formality. A clause that lands across a boundary is now two halves, and neither half answers anything.
Then every piece becomes a position
Each chunk is passed through a second, much smaller model — an embedding model — which turns it into a list of numbers. Here, 384 of them. That list is a position in a space with 384 axes, and the model is trained so that text with similar meaning lands in similar places. Not similar wording: similar meaning.
This is the move that makes the whole thing work. Meaning is hard to search. Geometry is easy to search. Once every paragraph is a point, “find the relevant passage” becomes “find the nearest points”, and that is arithmetic.
The question is a point in the same space
A question gets embedded exactly like a chunk — same model, same space. Then the system takes the chunks whose positions sit closest to it, and those are what the model is allowed to read.
The figure below does this on a fictional municipal software-service contract, cut into 18 clauses. Pick a question and watch where it lands.
Choose a question above to see which clauses it retrieves.
paraphrase-multilingual-MiniLM-L12-v2 — 384 dimensions, computed once when this page was built, so nothing is downloaded or run in your browser. Position is a PCA projection of the real embeddings onto two axes; those two carry 28% of the variance, so this picture is a flat shadow of the real space and two points that look close may not be. Colour is the section heading each clause was written under — a label the model never saw, shown so you can check its work. Ranking is always computed in the full 384 dimensions, never from the flattened positions drawn here, which is why the highlighted clauses can look slightly out of order.Read the plot as a shadow rather than a map. Two axes are standing in for 384, so the colour groups blur into each other far more on screen than they do in the space the model actually works in — the ranked list beside it is the honest view, because those numbers come from all 384 dimensions. What the picture is good for is showing that a question is the same kind of object as a paragraph, sitting in the same space, close to some things and far from others.
Two things in there are worth dwelling on.
Ask “what happens if customer data ends up in the wrong hands?” and the top result is the clause about reporting a data breach within 24 hours. The only word those two share is “data”. Nothing matches “wrong hands”, because the contract never says “wrong hands” — it says “breach of the protection of personal data”. A keyword search fails here outright, or drowns the right clause among every other paragraph containing the word “data”. The embedding finds it because the two phrases mean nearly the same thing, and nearness of meaning is exactly what the space encodes.
Ask how quickly an invoice has to be settled and the margin gets thin. The correct clause — payment within 30 days — comes first at 0.447. Second, at 0.435, is a clause about the contract tacitly extending by twelve months. It is the wrong answer, it is not even from the right section, and it lost by 0.012. Deadlines resemble deadlines. This is the ordinary condition of a retrieval system, not an edge case.
The score does not know whether the answer is there
Now the fourth question: how many deadlines in the contract are shorter than three months?
Retrieval answers confidently. Its top hit scores 0.605 — higher than the correct answer to the invoice question managed. And it is useless, because no single clause contains that answer. Answering it means reading all 18, finding every deadline, comparing each to three months, and counting. Retrieval is built to return the three nearest paragraphs. It cannot count, and nothing about a high score reveals that it has not.
This is the single most important thing to understand about similarity search: the score measures how much a chunk resembles the question, not whether it contains the answer. A system that treats “top hit above threshold” as “we have what we need” hands the model three plausible, irrelevant paragraphs and an instruction to answer — and the model, obligingly, invents the rest. What that costs, and the one rule that fixes most of it, is measured here.
Whole classes of ordinary business questions are shaped this way. How many contracts expire in Q3? Which supplier appears most often? Has this clause changed since the last version? None of them are retrieval problems, and all of them arrive phrased exactly like retrieval problems.
Two passes, not one
The fix for the near-misses is not a better embedding model. It is a second pass.
The reason the two stages differ in quality is worth stating plainly. In the first pass, the clause was embedded months ago, alone, with no idea what would be asked of it — the comparison is between two summaries made in ignorance of each other. A reranker reads the question and the clause at the same time, so it can notice that “settle an invoice” is about payment terms and that a twelve-month extension is a deadline of an entirely different kind. That is the 0.012 above, resolved.
You pay for it in time, which is why nobody runs it over the whole archive: cheap and wide first, expensive and narrow second.
Where this actually goes wrong
In practice, the failures are rarely exotic:
- Boundaries. A definition split across a chunk, a table separated from its heading, an answer that needs the sentence before it. See the first figure.
- Questions that need aggregation. Counting, comparing, “how often”, “which is the newest” — the shape from the fourth query. These need filters and structured queries, not vectors.
- In-house vocabulary. A general embedding model has never seen your internal abbreviations and has no idea that “BV-34” refers to a works agreement. It will place that chunk somewhere arbitrary and never contradict you about it.
- Compound words. German glues its nouns together. Kundendaten and Daten are one token apart for a person and unrelated strings for a keyword index — which is exactly why hybrid setups that lean on keyword search need real care in German.
- Two languages. A German question against an English document only works if the embedding model was trained for it. The one behind the figure above was; most of the popular defaults are English-only, and fail this silently.
- Scans and tables. A number in a cell loses its row and column heading the moment the table is flattened into a line of text, and a scanned page has no text at all until someone recognises it.
None of these are solved by picking a larger language model. They are solved before the language model is reached at all — in how documents are cut, what gets indexed alongside them, which questions are routed away from retrieval entirely, and what the system does when the best match simply is not good enough.
That last one is where trust is won or lost, and it is the subject of the next note.