retrieval visualized/ Ask About Analytics

Retrieval, visualized

A retrieval system built from parts, with every stage on screen.

Retrieval-augmented generation answers a question by searching a set of documents and quoting what it finds. This was built from parts. Each stage is drawn as it runs, each setting can be changed and the question re-run, and each default traces back to the measurement that chose it.

Overview

reading…
…
loading

Three document sets are searchable here, and they are fixed. They hold 36 Machine Learning papers, 45 Ornithology articles and 35 Quantitative Finance papers. Each one was indexed, given labelled questions and calibrated before any figure on this site was measured. The analytics page shows every configuration, every threshold sweep and every case.

The site is a learning project, built to understand retrieval by making every part of it visible rather than to serve documents at scale. To point it at your own documents, or to run the search live rather than replay a recorded one, run the project from GitHub. Every document in every set is listed there with its title and format, and every stage drawn on these pages is a function in that repository.

The usual way to build this is LangChain, LlamaIndex or Haystack, which supply the retriever, the pipeline and the defaults, so a working system is a short script. This uses none of them. It uses libraries instead, FAISS for the dense index, rank_bm25 for the sparse score and sentence-transformers to run the models, and the code between those stages is the project's own. What a framework gives up is the reason. Five settings here had to be measured for each set of documents rather than chosen once, and the three abstention thresholds that came out of it span 5.5 points, so no single default would have served all three.

Reading a result

Anatomy of an answer

To understand how to read a result, hover over each part below.

Reading…

Pipeline

Six stages, in order

  1. Chunking

    Each document is cut into overlapping windows of about 210 tokens. The encoder reads a bounded amount of text at once, which is a property of the model rather than a preference, and consecutive windows overlap by 40 tokens so that an answer sitting across a cut still appears whole in one of them.

  2. Hybrid retrieval

    Dense retrieval compares meaning, by turning the question and every passage into vectors. BM25 compares exact words and weights rare ones heavily. They run at the same moment. Each fails where the other works. Dense retrieval handles a question worded differently from its answer and misses an exact token such as a model number, because that token is averaged into the same vector as everything else in the passage and largely washes out. BM25 finds that token at once and returns nothing when the question and the answer share no words, so running both covers each gap.

  3. Fusion

    Each retriever contributes 1/(60 + rank) to every passage it returned, and the sums are sorted. Only the position counts, never the score itself, because the two retrievers measure different things on different scales and adding their numbers would require inventing a conversion between them.

  4. Reranking

    A cross-encoder scores the shortlist again by reading the question and the passage together and asking whether one answers the other. The earlier stages turned the question and the passage into numbers separately, so the two never met and the comparison could only score what a passage is about rather than whether it answers. Reading the pair is far more accurate and far slower, and it takes over 90% of the time a query spends, which is why it sees a shortlist rather than the corpus.

  5. Diversity cap

    At most two passages from any one document survive. The reranker scores each passage against the question on its own, so one document that matches well can take every slot. The cap walks the ranked list and skips a passage whose document is already full, keeping the skipped ones in reserve so five results still come back when few documents qualify. It is a trade, because it lowers the hit rate a little and raises how many of the answering documents come back, and a question answered by four papers should not return four passages from one of them.

  6. Abstention

    If the best passage scores below this corpus's threshold, the system declines instead of answering. A search always returns its closest matches, however far away, so the ornithology set asked about tax law returns five passages about birds. The threshold is set for each set of documents by measuring where its real answers stop and its near-misses begin. It catches a question the documents cannot answer, and it does not catch a wrong passage, because every question these sets get wrong scores above its own set's threshold.

Development

any-hit@5, Machine Learning

Each stage was added because the one before it was measured and found wanting. Every change below was written, run and scored before it was judged. Figures are any-hit@5 on the 36-paper set, the share of questions where a correct passage came back in the top five, unless the row labels them otherwise.

Kept

  1. Dense retrievalkept

    Every passage becomes a vector and the question is compared against all of them. This is the whole of what most tutorials build.

    any-hit 0.821

  2. Cross-encoder rerankingkept

    A second, slower model re-reads the shortlist one pair at a time. Comparing two separately made vectors is cheap and blunt, and reading the pair is expensive and much better at telling a passage that mentions the subject from one that answers the question.

    0.821 → 0.836

  3. BM25 beside dense retrievalkept

    Meaning-matching misses a term that carries its meaning by being exact, such as a model number or a species name. A second retriever scores word overlap and the two rankings are merged, which was the largest single gain in the project.

    0.836 → 0.925

  4. Diversity capa trade

    One thorough document could fill every slot and hide a second document that also answered. The cap costs hit rate and buys coverage, and it is the only change here that shipped while making a headline number worse.

    0.925 → 0.910, documents found 0.745 → 0.785

  5. The excerpt windowkept

    The excerpt once began at the start of the passage, and on a paper's first page that is the title and the authors' addresses, so a question about the paper's subject was answered on screen with its letterhead while the answering sentence sat just past the edge. It now opens at the sentence carrying most of the question's words and runs to 720 characters. What is found did not change, and what is shown did.

    answer on the page 0.656 → 0.808
Not kept, ranking4 changes

  1. A larger rerankerrejected

    A model twelve times the size recovered two of the ten failing questions, the same two a model 1.5 times the size reached, and took ten times as long. Across three models, ranking gets worse as they grow while telling answerable from unanswerable gets better.

  2. A quantised rerankerrejected

    Compressing the cross-encoder ran 1.5× faster and barely moved the rankings, which is all a normal comparison checks. Every score also shifted down a fifth of a point, and the gate reads that score as an absolute, so the system silently began refusing questions it could answer.

  3. A wider candidate poolrejected

    More candidates made the answer available more often and returned less often, on all three sets, because the reranker's precision falls faster than the extra candidates help. It is the clearest evidence in the project of where the weak stage is.

  4. A second embedderone set only

    Two models read the passages and their rankings are fused. Better on the shortlist everywhere, and on the finished answer worse on the papers, an even trade on the birds, and better only on finance, where it ships.

Not kept, the question4 changes

  1. Query rewritingrejected

    A language model restated each of the seven failing questions more clearly first. It reached one and broke five, because paraphrasing a question that already works moves it off the wording that was working.

    net three questions worse, and six times slower

  2. Query decompositionrejected

    Splitting each question into its parts and searching for each reached none of the seven and cost the same three. It had been the project's named answer for those seven for months, on an argument rather than a measurement.

    0 of 7, at 21× the time

  3. An invented answerrejected

    Four of the five unreachable questions ask for a word they do not contain, so a model was asked to answer from memory and the search run on its invented passage. It supplies the vocabulary while the documents still supply the citation, and run on every question it is worse.

    paper set +3 / −7

    Firing it only when the system was about to decline never triggers. Every one of the eleven questions these sets get wrong scores above its own set's refusal line, the highest at +6.36, so the system is not uncertain when it is wrong. Three cheaper signals were tried in the gate's place and all three put the failing questions inside the range of the working ones.

    reaches 0 of 11

  4. Pseudo-relevance feedbackrejected

    Adding words from the first results is the classical fix for a question whose answer uses different words. It recovered none of the twelve, for a reason written down before the run, because those words come from the passages the first search returned and the missing word is not in them either.

    0 of 12 recovered
Not kept, the neighbouring passage4 changes

  1. Neighbours on the pagerejected

    Showing the passages either side puts the answer on screen for 9 more questions and triples the text, or 4 questions for nearly twice the text once restricted to what the page renders. Choosing the excerpt from the enlarged passage scored worse than doing nothing.

    0.808 → 0.840, at 591 → 1,073 characters a passage

  2. Neighbours in the shortlistrejected

    Offering the neighbour to the reranker instead gains one question on one set, costs ranking quality on the other two, lets one more unanswerable question through the gate, and takes two and a half times as long.

    one question gained, MRR −0.012 and −0.015

  3. Window scoringrejected

    Ranking each passage by the piece it sits in, while still returning the piece itself, is worse on every set and every measure but one. It lets fewer unanswerable questions through only because scoring more text lowers every score at once, which moves the refusal line rather than the ranking.

    0.910 → 0.881 on the papers, 0.939 → 0.818 on finance

  4. A finer striderejected

    Cutting the documents so passages start closer together means 32% more of them to search. On the ornithology set it is worse at finding the answer and worse at showing it, and it recovers exactly the one question the shortlist experiment above recovers.

    0.880 → 0.840 found, 0.760 → 0.720 shown
Not kept, what is shown2 changes

  1. Title prefixesreverted

    Putting each document's title on every passage bought no ranking gain and lost coverage. An earlier version had looked like a win because two things were changed at once.

    documents found −0.051

  2. A front-matter filterleft off

    Dropping a paper's title block from the search scores worse on both sets that have title pages. Both passages behind the drop answer outright, because a passage that begins with a title page continues into the abstract, so the filter loses two real answers along with the title.

    0.851 → 0.836
Not kept, generation1 change

  1. A larger generatorno change

    A generator three times the size got two more of the 28 answerable questions right, which is inside the noise at this size, and everything else moved the other way. It misses one of the refusals, quotes reference numbers that name no document, draws less of its wording from the passages it was given, and takes 58 times as long.

Shared failure modes

Nine of the rejected changes aim at the same two dozen questions and fail for one reason. The passage that answers does not contain the words of the question, which is why the search missed it, so putting that passage in front of the ranker changes nothing, since the ranker reads the same question and reaches the same conclusion. Showing more of the document around it, cutting the document differently and having a model invent an answer to search with are all ways of offering the same passage again. A question that asks what something is called withholds the one word that would find it, and every part of this system that ranks, chooses or emphasises by the question's words moves away from the answer when given more room. That is a property of the question, not of the documents.

Four separate experiments improved what the first search finds and made the finished answer worse. Finding the answer and returning it are different problems, and a change that helps the first can cost the second. The clearest case is the second embedder, whose shortlist on the paper set is identical to the shipped one with only the order changed, and the reranker does worse with that order.

Limitations

Known weaknesses

The answer can be wrong, and it does not look wrong when it is. Across the three sets, eleven questions have nothing that answers them, and the system shows an answer anyway on nine of those, the highest scoring +6.36 on a scale reaching about +11.

Based on relevancy

The decision to answer or decline reads the cross-encoder's score, which rates the match between question and passage and not whether the passage is right. When the wrong passage scores higher, the system answers with confidence. Every question these sets get wrong scores above its own refusal line.

Vocabulary mismatch

When the passage that answers does not contain the words of the question, the search misses it. Putting that passage in front of the reranker does not help, because the reranker reads the same question and reaches the same conclusion. A question that asks what something is called withholds the one word that would find its answer.

Sample size

Every figure on the analytics page comes from 157 labelled questions across the three sets. At that size a difference of one or two questions is inside the noise, so a small gain between two configurations is not evidence that one is better.

Glossary

Terms used in retrieval
Terms21 entries
RAG
Retrieval-augmented generation. Answering a question by searching a set of documents first and quoting what is found, rather than relying on what a model memorised. retrieval-augmented generation
Corpus
One set of documents with everything derived from it, which is its index, its labelled questions and its threshold. Nothing about one corpus transfers to another, and that is the finding this project has reused most. document set
Chunk
One passage, a window of about 210 tokens cut from a document with 40 tokens of overlap onto its neighbour. passage
Locator
Where a passage came from, in the unit its format has, which is a page for a PDF, a slide for a deck, and a sheet or section elsewhere. Saying “page 3” of a slide deck sends a reader somewhere that does not exist.
Embedding
A passage or question turned into a list of numbers, positioned so that things meaning similar things sit near each other. vector
Dense retrieval
Search by comparing embeddings. It matches meaning and handles a question worded differently from its answer, and it misses a rare token that carries its meaning by being exact, such as a model number. vector search · semantic search
BM25
Search by exact term overlap, weighting rare words far above common ones. Its profile is the inverse of dense retrieval, strong on identifiers and blind to paraphrase. sparse retrieval · lexical search
Fusion
Combining two rankings into one. This system uses reciprocal rank fusion, in which each retriever contributes 1/(60 + rank), so only position counts and no score conversion is needed. RRF · hybrid search
Candidate pool
The shortlist the first stage hands to the reranker. Its size is a latency dial, because reranking is over 90% of the time a query takes and the cost scales with the pool. candidate_k
Cross-encoder
A model that reads a question and a passage together and scores whether one answers the other. It is far more accurate than comparing two separately made embeddings and far too slow to run over a whole corpus. reranker
Reranking
Re-ordering the shortlist with the cross-encoder. On some corpora a fraction of the first stage's ordering is deliberately kept. When a question describes a term instead of naming it, the cross-encoder has little to work with.
Diversity cap
A limit of two passages per document in the final answer. It trades a little hit rate for covering more of the documents that answer the question.
Context expansion
Attaching the neighbouring passages to each result so a sentence cut by a chunk boundary is still readable. It cannot change any ranking, because it runs after selection. It is on for the generator and off for the page, which was measured separately.
Abstention threshold
The score below which the system declines to answer, calibrated per corpus, because the same number that costs one corpus a single question costs another ten. It separates answerable questions from unanswerable ones rather than right passages from wrong ones. the gate · confidence threshold
Golden set
The labelled questions a corpus is scored against, each with the documents and locations that answer it. Every number on the analytics page comes from these.
Adversarial case
A question included because the documents cannot answer it. Refusing one is a success, and answering it is the failure being measured. unanswerable case
any-hit@5
The fraction of questions where at least one passage from a location that answers appears in the five returned. It asks whether the right place was reached.
Answer shown
The fraction of questions where the answer itself is in the text returned. It is lower than any-hit, and the gap is the point, since a hit credits the right page and a page can hold the answer in a paragraph that was not returned. answer_visible
MRR
Mean reciprocal rank. It rewards putting the right passage first, scoring 1.0 if the first passage is correct, 0.5 if the second is, and so on. mean reciprocal rank
NDCG
A ranking-quality score over all the relevant passages returned, not only the first, discounting each by how far down it sits. normalised discounted cumulative gain
Source recall
How many of the documents that answer a question were returned. Hit rate cannot express this, because returning one of four answering papers scores a perfect hit rate while missing three quarters of the answer.

Built by hand in Python, with no retrieval framework.