SugarCrepe: Test Whether AI Can Tell Who Did What in an Image

A shop has two photographs of the same pair of mugs. In one, the blue mug sits on the yellow tray. In the other, the yellow mug sits on the blue tray. An image search system retrieves both for the phrase ‘blue mug on a yellow tray’. It has found the right objects and colours. Only one photograph has the right arrangement.

That difference can matter when you use AI to find product photos, check draft captions or prepare instructions. A system may recognise everything in the frame and still put an attribute on the wrong object or reverse who did what. The question is how to test that skill without accidentally giving away the answer in the wording.

Make both captions sound like something a person would say

In the 2023 SugarCrepe paper, Cheng-Yu Hsieh and colleagues examined tests that ask a model to pick which of two captions fits an image. They found that some earlier tests contained clues that did not require looking at the picture. For example, a deliberately incorrect caption might sound less fluent or less plausible than the correct one. Models that saw only the text could perform surprisingly well on the benchmarks the authors examined.

That is a problem for interpretation. If the model can choose the smoother sentence, a good score may partly reflect sensitivity to language rather than the ability to match a sentence to a scene. Imagine asking a model to choose between ‘a baker puts a loaf on a rack’ and ‘a rack puts a loaf on a baker’. The grammar is similar, but the second caption is so implausible that the image barely matters. A better pair might say that the baker puts the loaf on the rack or takes it off. To pick between those, you need evidence about the action.

The SugarCrepe team used a language model to draft plausible alternatives, then used human checks and a refinement step to reduce the text-only clues they had identified. Their project repository describes the resulting task: given an image, choose the caption that fits it over a near match. The researchers reported that the pretrained CLIP models they evaluated struggled with swapped elements. That finding is about the models and test they used in 2023. It does not establish how a particular tool available today will perform on your photos.

Build a small test around mistakes that matter

Start with the job you want the tool to do. If it drafts shop captions, a wrong colour assignment can misdescribe a variant. If it searches an image archive, a missed distinction can send you the wrong photograph. If it describes a tutorial, reversing ‘put on’ and ‘take off’ may change the instruction. Choose examples where the detail changes what somebody would select or do.

For a test like the mug example, photograph both arrangements if you can. Keep the two captions unchanged and give the model each photograph in turn. The correct answer should change with the image. For another pair, photograph a small box beside a larger one, then put the small box inside it. Both descriptions should be natural and plausible. Avoid using a photo where the relevant detail is hidden, unless the purpose is to test whether the tool will admit it cannot tell.

Before running the image test, read the caption pairs without seeing the photographs. If one sounds much more natural, rewrite it. Vary which answer comes first, and keep the wording and level of detail reasonably balanced. This does not remove every possible clue, but it helps you avoid congratulating the system for passing a test you made easy for the wrong reason.

Measure the job you will actually rely on

A forced choice is a useful diagnostic because it isolates a particular distinction. It is not the same as asking for an original caption, searching a real archive or dealing with a picture where neither caption fits. Check the diagnostic first, then try the real workflow separately. Record whether the correct image was retrieved, whether the model described the decisive detail accurately, and whether the output makes sense to a reader who has not seen your test notes.

• Check the failures by type. Group mistakes such as reversed actions, misplaced attributes and uncertain details. A single score can conceal a recurring error on the cases you most need to get right.
• Compare against a simple alternative. If a catalogue already stores colour and product code, see whether an ordinary field lookup settles the exact part of the task more reliably. You can still use image search to find visually similar items.
• Keep a human review where the error has consequences. If a wrong relation changes a product claim or a tutorial instruction, check the image before publishing. Let the draft flag uncertain details for review instead of insisting on an answer.

For retrieval, check where the error occurred. A strong caption scorer cannot pick the right photograph if the first pass never retrieved it. The MaryChuks guide to testing a search shortlist before reranking explains how to separate those stages.

The next time an image tool passes a demo, change one relationship and leave everything else alone. If the answer should change but does not, you have found a specific limit worth designing around. That tells you more about the job than a gallery of plausible captions.

Sources

• Hsieh et al., SugarCrepe: Fixing Hackable Benchmarks for Vision-Language Compositionality, 2023.
• SugarCrepe benchmark: authors’ project repository and evaluation notes.