Merlin and CT Scans: Finding the Matching Report Is a Test of Its Own

Put a scan on one side of a desk and a stack of reports on the other. Ask a system to find the report written for that scan. Now remove the original report and ask it to find a useful precedent. Finally, ask it to write a fresh account of what the scan shows. These requests sound like three versions of the same task. They are three different tests, and a system can do well on one without giving you the evidence you need for the next.

That distinction helps make sense of Merlin, a research model that works with volumetric CT scans and text. It also matters well outside medicine. Anyone who has tried to connect photographs with archive notes, or product images with records, has to decide what a match means before deciding whether an AI search tool works.

What the study actually paired

In the study published online in Nature on 4 March 2026, Louis Blankemeier and colleagues trained Merlin using three kinds of material linked to CT examinations: scan volumes, radiology reports and diagnosis codes. A volume contains a series of image slices, giving the model an input with depth. The team used existing clinical records in training without commissioning extra manual annotations for that stage.

The presence of a report beside an image is useful information, but the two records have different jobs. One supplies the image data; the other records what someone documented about the examination. The relationship between them gives the model a way to learn which kinds of text and image tend to go together. It does not mean that every sentence in a report describes something visible in the pixels or that every visible detail was written down.

Finding the original is a defined problem

The paper’s retrieval experiment asked whether the model could find the matching scan or the corresponding report section among candidates, including an evaluation with pools of 64 cases. The researchers tested report findings and impressions as distinct text inputs. In the scan to findings comparison, Merlin performed better than the OpenCLIP and BioMedCLIP baselines the authors tested. The task is called cross-modal retrieval because the query and the item being sought can be in different forms: an image can lead to text, and text can lead to an image.

This gives the experiment a clear answer key. The source record tells the researchers which report belongs to which scan. The result shows how well the model ranked the recorded counterpart under those test conditions. Read the result with the candidate pool in mind. A search where most candidates are obviously different asks less of a system than one full of near matches. The size of the pool is worth knowing, but the kinds of alternatives in it matter as well.

A search for a useful precedent has a different answer key. Several reports might be useful, or none might be close enough. The best example might share one important feature with the query while differing in several others. What counts as useful depends on what the person is trying to compare. This is a reason to define and test relevance separately, rather than treating a benchmark for finding the original record as proof that the system can always choose a suitable precedent.

Writing a new report raises a different question

Merlin’s researchers also evaluated report generation as a separate task after adapting the model for it. That distinction deserves attention. When a system retrieves an existing document, a reviewer can inspect the document it found and check its source. When it writes a new statement, someone must check whether the statement is supported by the underlying material. A document can read fluently and still miss a detail, turn uncertainty into a firm statement or include something that belongs to another case. The retrieval result alone cannot settle those questions or show how a system would perform in everyday clinical use.

Our article on reading Aurora’s air pollution forecast at the right scale makes a related point about evidence. A benchmark is informative when the question you ask it matches the question it was designed to answer. Here the boundary is between finding the recorded partner, finding a helpful comparison and making a new claim.

Try the distinction on a small archive

Suppose a design studio wants to search its collection of prototype photographs and workshop notes. Start by asking the tool to link a photograph of a lamp to the note written for that prototype. Include photos of the next revision, especially if the two versions look almost identical. Use the studio’s job records to settle which note belongs with which photo. That tests identification.

Next remove the original note and ask for another prototype that used a similar hinge. The best example may be an entirely different product. Let a person who understands the collection label acceptable results and note cases where nothing qualifies. Finally, ask the tool to describe the hinge in the photograph. Have a reviewer distinguish visible features from guesses about materials, dimensions or how it was assembled. Those three checks may lead to three different decisions about what the tool should do.

The useful lesson from Merlin is to give each job a test of its own. A system that reliably returns a record can save someone the work of finding it. Deciding whether that record is a good comparison, or whether a new description is accurate, takes evidence suited to that decision.

Source

• Blankemeier and colleagues (2026), Merlin: A Computed Tomography Vision-Language Foundation Model and Dataset, Nature; see section 2.3 for the retrieval evaluation.