When Two Searches Disagree: How Reciprocal Rank Fusion Combines Their Results

Imagine you sell replacement parts for desk lamps. A customer searches for ‘bracket for the short arm on a Mica L12’. The keyword search puts the L12 instruction sheet first because it repeats the model code. A search based on meaning puts a bracket for the longer arm first because its description sounds like what the customer wants. The correct short-arm bracket comes second in both lists. Which item should the customer see first?

You could blend the two search scores, but a keyword score and a similarity score need not have comparable scales or meanings. Reciprocal rank fusion, or RRF, takes a different approach: it combines the positions in the lists. That lets agreement lift a result that neither search placed first. It can also lift the wrong item when both searches make a similar mistake.

What the method counts

Gordon Cormack, Charles Clarke and Stefan Büttcher described RRF in a 2009 SIGIR paper. It is a search ranking method that predates today’s generative AI. The authors reported improvements in the retrieval experiments they ran. That is evidence for the method in those settings; whether it improves a particular shop, archive or help centre remains a question for that collection.

For every list in which a document appears, add a contribution of one divided by the sum of its position and a positive constant. Positions start at one. A document that is absent contributes nothing from that list. Sort by the total. Elastic’s reference for reciprocal rank fusion illustrates the rule and describes a setting that controls how many results each search can contribute.

Take the lamp example and use a constant of 10 just to make the arithmetic easy to follow. The bracket in second place in both lists scores 1/12 + 1/12, or about 0.167. The instruction sheet in first place in the keyword list, but absent from the other list, scores 1/11, or about 0.091. If the long-arm bracket appears only in first place in the other list, it also scores about 0.091. The short-arm bracket rises because two lists support it.

The constant affects how much first place matters relative to lower positions. A larger constant makes the contributions from nearby ranks more similar, giving appearances across lists more room to matter. The method also discards the size of any gap in the original scores. A runaway first place and a narrow first place contribute the same amount at that position. That is the price of combining ranks without comparing the original numbers.

Where agreement can mislead

Now imagine a different query: ‘Mica L12-B7’, an exact catalogue identifier. The correct part is first in the keyword list and absent from the semantic results. A popular installation guide appears second in both. The same maths can put the guide above the exact part. That is coherent ranking behaviour, but it is poor service if the shopper entered the code specifically to reach that item.

Treat exact identifiers as a separate case. For example, when a query cleanly matches a unique catalogue code, show the record prominently and use the blended list for related material. If an identifier is shared or ambiguous, present the candidates with a distinguishing detail such as the product line. Use the same principle in a help centre: distinguish a model-specific safety notice or an instruction for the right edition from a general article that both systems happen to favour.

A second limitation is the input window. If the right item ranks beyond the number of results you take from each search, fusion never sees it. This is related to the shortlist problem in our guide to Matryoshka embeddings: once a candidate has dropped out, a later ranking step cannot rescue it. Look at what enters the window before adjusting how the final list is sorted.

Run a small comparison before switching

Collect actual queries if you have permission to use them, or write representative ones with someone who handles customer questions. Decide which records would genuinely help before looking at the ranked outputs. For some queries one record is clearly right; for others several results may be useful. Keep those judgements visible when you compare keyword search, semantic search and the blended ranking.

• Exact lookups: Try identifiers, titles and edition numbers. Check how often the exact item is first and inspect cases where a popular guide beats it.
• Everyday wording: Include the ways customers describe a problem when they do not know your terms. Look for a useful answer in the first few results and note which search found it.
• Confusable results: Try products with adjacent model numbers, outdated guidance or similarly named services. Check whether shared mistakes are pushed higher when you combine the lists.

For a missed answer, record whether it appeared in either input list and at what position. If it appeared in neither, investigate coverage or the source searches. If it appeared but fell after fusion, look at the competing results and decide whether the query needs a different rule. Compare results by query type so improvements on broad searches do not hide regressions on exact ones.

A successful trial may justify blending only some searches. That is a useful outcome. The point of combining the lists is to help a reader find the right thing, and what counts as ‘right’ depends on whether they are exploring a topic or asking for one particular record.

Sources

• Cormack, Clarke and Büttcher, original reciprocal rank fusion paper, SIGIR, July 2009.
• Elastic, Elasticsearch reciprocal rank fusion reference, living documentation.