Retrieval, visualized
Retrieval-augmented generation answers a question by searching a set of documents and quoting what it finds. A framework builds one in an afternoon and the result works, but every decision it makes is hidden inside it. This one is built from parts, so each stage is drawn as it runs, each setting can be changed and the question re-run, and each default can be traced to the measurement that chose it over the alternatives.
At a glance
read live from the running systemThree prepared document sets are searchable here and they are fixed: 36 machine-learning papers, 45 ornithology articles and 35 quantitative-finance papers. Each set was indexed, given labelled questions and calibrated before any figure on this site was measured, and the figures above are read from the same generated file the analytics page plots, so this page cannot quote a number the measurements do not support. The analytics page shows every configuration, every threshold sweep and every case.
The site is a learning project, built to understand retrieval by making every part of it visible rather than to serve documents at scale. To point it at your own documents, or to run the search live rather than replay a recorded one, run the project from GitHub. Every document in every set is listed there with its title and format, and every stage drawn on these pages is a function in that repository.
Compared with a closed product
Five points of differenceNotebookLM, Glean and the enterprise search tools sold to companies answer better than this, because they have larger models and far more compute. They also answer from a closed middle, where a question goes in, an answer comes out, and what happened between the two is not shown.
Reading a result
The front pagePick a document set, type a question, and the answer comes back as the passage it rests on together with the route it took. Nothing is generated by default, so what you read is the retrieved text word for word.
The passage is shown verbatim, with only the words that answer set in bold. The bold is a pointer chosen by overlap with the question, and a question that asks what something is called withholds the one word that would be pointed at, so on those the passage carries no bold at all. The citation names the document's real title and the location in the unit its format has, which is a page for a PDF, a slide for a deck and a sheet for a spreadsheet.
The score comes from the cross-encoder and runs roughly −11 to +11. It is neither a probability nor a percentage, and it does not say whether the passage is right. It says how strongly the model matched this passage to this question, which is a different quantity. Across the three sets there are eleven questions where retrieval returns nothing correct, and nine of them score positive, the highest at +6.36. A strongly negative score does mean the documents probably do not hold the answer. A high one does not mean they do.
“Also found” lists the other passages that came back, one from each further document first and then the rest in the order they were ranked. The top passage is sometimes the wrong one, and on 27 of the 121 recorded answers that carry a labelled answer string, the passage holding the answer sits in this list rather than at the top. Asked for a bird's fused collarbone, the cross-encoder ranks the pygostyle passage first and the furcula second.
“The corpus does not contain this” means the gate declined, which is the system working. Some of the example questions on the front page are chosen to trigger it.
A passage rather than a written answer. Everything above is text taken from a document. Composing an answer is the step where a system can state something no document said, so although this one can write an answer from the same passages with a model on the serving machine, that is off here, because the site exists to show the retrieval. When it does compose, the result is scored on whether it cites a document it was never given, whether it declines the questions the documents cannot answer, and how much of its wording came from the passages. No model judges those answers, since a judge is a second system whose own mistakes are invisible. The figures are on the analytics page.
Pipeline
Six stages, in order-
Chunking
Each document is cut into overlapping windows of about 210 tokens. A whole document is too large to compare against a single question, and the windows overlap so that an answer sitting across a cut still appears whole in one of them.
-
Two retrievers
Dense retrieval compares meaning, by turning the question and every passage into vectors. BM25 compares exact words and weights rare ones heavily. They run at the same moment. Each fails where the other works. Dense retrieval handles a question worded differently from its answer and misses an exact token such as a model number, and BM25 finds that token at once and returns nothing when the question and the answer share no words. Running both covers each gap.
-
Fusion
Each retriever contributes
1/(60 + rank)to every passage it returned, and the sums are sorted. Only the position counts, never the score itself, because the two retrievers measure different things on different scales and adding their numbers would require inventing a conversion between them. -
Reranking
A cross-encoder scores the shortlist again by reading the question and the passage together and asking whether one answers the other. The earlier stages turned the question and the passage into numbers separately and compared the numbers. Reading the pair is more accurate and much slower, which is why it only sees a shortlist.
-
Diversity cap
At most two passages from any one document survive. This is a trade. It lowers the hit rate a little and raises how many of the answering documents come back, because a question answered by four papers should not return four passages from one of them.
-
The gate
If the best passage scores below this corpus's threshold, the system declines instead of answering. A search always returns its closest matches, however far away, so the ornithology set asked about tax law returns five passages about birds. The threshold is set for each set of documents by measuring where its real answers stop and its near-misses begin. It catches a question the documents cannot answer, and it does not catch a wrong passage, because every question these sets get wrong scores above its own set's threshold.
Development
any-hit@5, ML papersNothing here was designed in one pass. Each stage was added because the one before it was measured and found wanting, and every change below, kept or not, was written, run and scored before it was judged. The figure beside a kept change is, unless the row labels it otherwise, the share of questions where a correct passage came back in the top five on the paper set.
Kept
-
Dense retrievalkept
Every passage becomes a vector and the question is compared against all of them. This is the whole of what most tutorials build. any-hit 0.821
-
Cross-encoder rerankingkept
A second, slower model re-reads the shortlist one pair at a time. Comparing two separately made vectors is cheap and blunt, and reading the pair is expensive and much better at telling a passage that mentions the subject from one that answers the question. 0.821 → 0.836
-
BM25 beside dense retrievalkept
Meaning-matching misses a term that carries its meaning by being exact, such as a model number or a species name. A second retriever scores word overlap and the two rankings are merged, which was the largest single gain in the project. 0.836 → 0.925
-
Diversity capa trade
One thorough document could fill every slot and hide a second document that also answered. The cap costs hit rate and buys coverage, and it is the only change here that shipped while making a headline number worse. 0.925 → 0.910, documents found 0.745 → 0.785
-
The excerpt windowkept
The excerpt once began at the start of the passage, and on a paper's first page that is the title and the authors' addresses, so a question about the paper's subject was answered on screen with its letterhead while the answering sentence sat just past the edge. It now opens at the sentence carrying most of the question's words and runs to 720 characters. What is found did not change, and what is shown did. answer on the page 0.656 → 0.808
Rejected, ranking4 changes
-
A larger rerankerrejected
The obvious answer to the reranker being the weakest stage was a model twelve times its size. It recovered two of the ten failing questions, the same two that a model 1.5 times the size recovered, and took ten times as long per query. Three models were compared, and the two halves of the job move in opposite directions as the model grows, since the larger ones rank worse and separate answerable from unanswerable better.
-
A quantised rerankerrejected
Compressing the cross-encoder ran 1.5× faster and barely moved the rankings, which is all a normal comparison would have checked. Every score also shifted down by a fifth of a point, and the gate reads that score as an absolute, so the system silently began refusing questions it could answer.
-
A wider candidate poolrejected
More candidates means the answer is available more often, and it was returned less often on all three document sets, because the reranker's precision falls faster than the extra candidates help. This is the clearest statement in the project of where the weak stage is, and it came out of an experiment about speed.
-
A second embedderone set only
Two different models read the passages and their results are combined. On a candidate shortlist it looked better on all three document sets, and measured on the finished answer it is worse on the papers, an even trade on the birds, and better only on the finance set, where it ships. On the papers the shortlist does not change at all, since the same passages arrive in a different order and the reranker does worse with that order.
Rejected, the question4 changes
-
Query rewritingrejected
Seven questions on the paper set fail however the search is configured, and the plan for them was to have a language model restate each one more clearly first. It reached one of the seven and broke five that had been working, because a rewrite is a paraphrase and paraphrasing a question that already works moves it off the wording that was working. net three questions worse, and six times slower
-
Query decompositionrejected
For the same seven, a second plan split each question into its parts, searched for each part and combined the results. It reached none of them, cost the same three questions, and took 21 times as long per search. It had been the project's named answer for those seven for months, on an argument rather than a measurement. 0 of 7, at 21× the time
-
An invented answerrejected
Of the five questions no configuration reaches, four ask for a word the question does not contain, so the word which would find the passage is absent from the search. A model was therefore asked to answer from memory and the search was run on its invented passage, which supplies the vocabulary while the documents still supply the citation. Run on every question it is worse. On the paper set it recovers three and breaks seven, and on the finance set it recovers none and breaks four. paper set +3 / −7
Running it only when the system was already about to decline, so it could not disturb a working question, was measured before it was built and never fires, because every one of the eleven questions these sets get wrong scores above its own set's refusal line, the highest at +6.36. The system is not uncertain when it is wrong. Three cheaper signals were tried in the gate's place, which are how many of the question's words the top passage carries, how far the top score leads the fifth, and how many of the top five both retrievers ranked, and on all three the failing questions sit inside the range of the working ones, so catching all eleven means firing on at least 90% of every question. reaches 0 of 11
-
Pseudo-relevance feedbackrejected
Expanding the query with words from the first results is the classical fix for a question whose answer uses different words. It recovered none of the twelve questions it was aimed at, for a reason written down before the run, which is that the extra words come from the passages the first search returned and the missing word is not in those either. 0 of 12 recovered
Rejected, the neighbouring passage4 changes
-
Neighbours on the pagerejected
When the answer is missing from what is shown it is usually one passage away, so the obvious repair is to show more. Shown in full, the neighbours put the answer on screen for 9 more questions and triple the text. Restricted to what the page actually renders, they are 4 questions for nearly twice the text. Choosing the excerpt from the enlarged passage instead scored worse than doing nothing, 0.792 against 0.808, because the choice follows the question's words and moves further from the answer when given more room. 0.808 → 0.840, at 591 → 1,073 characters a passage
-
Neighbours in the shortlistrejected
Rather than show the neighbour, offer it to the model that ranks, so the piece holding the answer sits in front of the reranker beside the piece that was found. It gains one question on one set of documents, costs ranking quality on the other two, lets one more unanswerable question through the gate, and takes two and a half times as long. one question gained, MRR −0.012 and −0.015
-
Window scoringrejected
The same idea approached from the other end ranks each passage by the piece it sits in while still returning the piece itself. It is worse on every set of documents and on every measure but one. It does let fewer unanswerable questions through, and that is not an improvement, because scoring against more text lowers every score at once and moves the refusal line rather than the ranking, at a cost of 7 answerable questions on the papers. 0.910 → 0.881 on the papers, 0.939 → 0.818 on finance
-
A finer striderejected
If no single passage holds both the words that find it and the words that answer, cut the documents so that one does. The pieces stay the same length and start closer together, which is 32% more of them to search. Measured on the ornithology set with the shortlist widened to match, it is worse at finding the answer and worse at showing it, and it recovers exactly the one question the shortlist experiment above recovers. 0.880 → 0.840 found, 0.760 → 0.720 shown
Rejected, what is shown2 changes
-
Title prefixesreverted
Putting each document's title on every passage should have helped a passage say what it belongs to. It bought no ranking gain and lost coverage, and an earlier version had looked like a win because two things were changed at once. documents found −0.051
-
A front-matter filterleft off
Dropping a paper's title block from the search was the obvious repair for the excerpt problem, written before the cause was understood. It scores worse on both sets that have title pages, and the first reading was that the scoring had credited a title page which answers nothing. Read in full, both passages behind the drop answer outright, because a passage that begins with a title page continues into the abstract, which states the paper's finding in two or three sentences. The filter loses two real answers by dropping the abstract with the title, and judging a passage by how it opens is the error that put a title page on screen in the first place. 0.851 → 0.836
Generation1 change
-
A larger generatorno change
A generator roughly three times the size, from the same family, was compared over the 43 questions both models answered. It gets two more of the 28 answerable ones right, which on a set this size is inside the noise, and everything else moves the other way. It misses one of the questions it should refuse, it quotes the papers' own reference numbers, which name no document and cannot be looked up, it draws less of its wording from the passages it was given, and it takes 58 times as long.
The shared cause
Nine of the rejected changes aim at the same two dozen questions and fail for one reason. The passage that answers does not contain the words of the question, which is why the search missed it, so putting that passage in front of the ranker changes nothing, since the ranker reads the same question and reaches the same conclusion. Showing more of the document around it, cutting the document differently and having a model invent an answer to search with are all ways of offering the same passage again. A question that asks what something is called withholds the one word that would find it, and every part of this system that ranks, chooses or emphasises by the question's words moves away from the answer when given more room. That is a property of the question, not of the documents.
Four separate experiments improved what the first search finds and made the finished answer worse. Finding the answer and returning it are different problems, and a change that helps the first can cost the second. The clearest case is the second embedder, whose shortlist on the paper set is identical to the shipped one with only the order changed, and the reranker does worse with that order.
Glossary
Terms used in this interfaceTerms21 entries
- RAG
- Retrieval-augmented generation. Answering a question by searching a set of documents first and quoting what is found, rather than relying on what a model memorised. retrieval-augmented generation
- Corpus
- One set of documents with everything derived from it, which is its index, its labelled questions and its threshold. Nothing about one corpus transfers to another, and that is the finding this project has reused most. document set
- Chunk
- One passage, a window of about 210 tokens cut from a document with 40 tokens of overlap onto its neighbour. passage
- Locator
- Where a passage came from, in the unit its format has, which is a page for a PDF, a slide for a deck, and a sheet or section elsewhere. Saying “page 3” of a slide deck sends a reader somewhere that does not exist.
- Embedding
- A passage or question turned into a list of numbers, positioned so that things meaning similar things sit near each other. vector
- Dense retrieval
- Search by comparing embeddings. It matches meaning and handles a question worded differently from its answer, and it misses a rare token that carries its meaning by being exact, such as a model number. vector search · semantic search
- BM25
- Search by exact term overlap, weighting rare words far above common ones. Its profile is the inverse of dense retrieval, strong on identifiers and blind to paraphrase. sparse retrieval · lexical search
- Fusion
- Combining two rankings into one. This system uses reciprocal rank
fusion, in which each retriever contributes
1/(60 + rank), so only position counts and no score conversion is needed. RRF · hybrid search - Candidate pool
- The shortlist the first stage hands to the reranker. Its size is a latency dial, because reranking is over 90% of the time a query takes and the cost scales with the pool. candidate_k
- Cross-encoder
- A model that reads a question and a passage together and scores whether one answers the other. It is far more accurate than comparing two separately made embeddings and far too slow to run over a whole corpus. reranker
- Reranking
- Re-ordering the shortlist with the cross-encoder. On some corpora a fraction of the first stage's ordering is deliberately kept. When a question describes a term instead of naming it, the cross-encoder has little to work with.
- Diversity cap
- A limit of two passages per document in the final answer. It trades a little hit rate for covering more of the documents that answer the question.
- Context expansion
- Attaching the neighbouring passages to each result so a sentence cut by a chunk boundary is still readable. It cannot change any ranking, because it runs after selection. It is on for the generator and off for the page, which was measured separately.
- Abstention threshold
- The score below which the system declines to answer, calibrated per corpus, because the same number that costs one corpus a single question costs another ten. It separates answerable questions from unanswerable ones rather than right passages from wrong ones. the gate · confidence threshold
- Golden set
- The labelled questions a corpus is scored against, each with the documents and locations that answer it. Every number on the analytics page comes from these.
- Adversarial case
- A question included because the documents cannot answer it. Refusing one is a success, and answering it is the failure being measured. unanswerable case
- any-hit@5
- The fraction of questions where at least one passage from a location that answers appears in the five returned. It asks whether the right place was reached.
- Answer shown
- The fraction of questions where the answer itself is in the text returned. It is lower than any-hit, and the gap is the point, since a hit credits the right page and a page can hold the answer in a paragraph that was not returned. answer_visible
- MRR
- Mean reciprocal rank. It rewards putting the right passage first, scoring 1.0 if the first passage is correct, 0.5 if the second is, and so on. mean reciprocal rank
- NDCG
- A ranking-quality score over all the relevant passages returned, not only the first, discounting each by how far down it sits. normalised discounted cumulative gain
- Source recall
- How many of the documents that answer a question were returned. Hit rate cannot express this, because returning one of four answering papers scores a perfect hit rate while missing three quarters of the answer.
Built by hand in Python, with no retrieval framework, no CDN and no external request from this page. The server binds to localhost, since the documents may be private, and a page about private documents should not call a font host, so the fonts are bundled.