Retrieval analytics

Each row is the row above it plus one more mechanism, run on the same questions, and the cross-encoder gains the most. The diversity cap is the only row that gives something up, accepting a lower hit rate in return for finding more of the documents that answer the question, and this system takes that trade.

Table

The pipeline was tuned on the ML papers. Two further corpora were built to find out how much of it is a property of the system and how much of the documents it was tuned on.

Do not compare source recall across these three corpora. Questions on the finance set are answered by more documents than questions on the ML set, so the finance corpus has to return more documents in the same five slots to earn the same score. Its divisor is larger. The scores are honest within a corpus and misleading between them.

Every other figure on this page counts a question answered when the answer is anywhere in the five returned. "Answer first" counts it only when the passage shown first is one that answers, which is what a reader meets before scrolling. The gap between the two is the ordering cost, and it is large: across the three sets 83 of 125 questions open with a passage that answers and 114 of 125 hold one somewhere. Asked "which passage ranking dataset is used for evaluation", the system returns the ColBERT passage naming MS MARCO second and an appendix about a reading-comprehension set first. The hit rate calls that a hit and the reader has to read past a wrong answer to reach the right one.

What the scoring could get wrong here, and what it does. A question counts as answered when a passage comes back from a page the labels mark correct. A paper's title page is page one, and a paper is usually titled after the thing it is about, so page one is a marked page for 43 of the 100 questions asked of the two document sets that have title pages. Counted at the shipped settings, two hits are satisfied only by a passage that opens with a title page, and both of those passages contain the answer, because what follows a title page is the abstract. The number of hits credited to a page that answers nothing is zero. The reach is real and the cost today is nothing, and only counting the cases tells you which.

Every question scored by the cross-encoder, plotted on one axis. Filled marks are questions the corpus can answer; hollow rings are adversarial questions whose answer is deliberately absent. The gate refuses everything left of its cut point, so the best threshold sits in the gap between the two groups. That gap is in a different place on every corpus, which is why one number cannot serve all three.

Moving each corpus to its own cut point recovers answers at no adversarial cost: the same number of planted questions is still caught in every case. It is a strictly dominant change, and it is visible only because the corpora were measured separately.

Counted in cases, not rates. Rates are the wrong unit here: the two populations are different sizes, so subtracting one rate from the other quietly values a single adversarial catch at several answerable questions thrown away.

Every other number on this page measures retrieval: which passages came back, and in what order. These measure the prose written from them. There is no model judging these answers. A judge model is a second system whose own failures are invisible, and it would make the claim that every number here is reproducible from the repository false. Citations and refusals are checked mechanically instead.