AI Image Segmentation: Decide What the Outline Represents

A neat outline around an object can make an AI result feel settled. The bird has been separated from the table. The background can disappear. A coloured overlay appears to tell us exactly what the machine has found.

But an outline answers a narrower question than its appearance suggests. Does it mark the whole bird, its detachable wing, its shadow, or one visible part of an object hidden behind another? And if several outlines appear, do they represent several physical items?

For independent creators and small businesses, these questions matter when turning photographs into product cut-outs, catalogues or counts. Image segmentation is useful work. Its usefulness grows when the person using it states what the selected region is supposed to mean.

What the original Segment Anything research established

In the 2023 Segment Anything paper, Alexander Kirillov and colleagues presented a promptable segmentation model, SAM. A mask identifies a region of an image. Points or a box can indicate the region a user wants selected.

The paper explicitly addresses ambiguity: one prompt can reasonably refer to a whole object, a part or a smaller subpart. SAM can produce multiple candidate masks rather than forcing every ambiguous prompt into one answer. During the researchers’ assisted annotation stage, object names and descriptions were not collected alongside the masks.

This establishes a useful distinction. Selecting pixels and assigning a name to what those pixels depict are different tasks. The original research should not be read as a promise that every outline automatically arrives with the right product identity, category or counting rule. This article concerns those research distinctions; it is not a review of the latest commercial segmentation tools.

A fictional workshop photograph

Imagine a maker preparing a catalogue photograph of a wooden bird ornament. The bird has a removable wing and rests on a small display stand. Another bird sits partly behind it. The maker wants an image of one complete saleable item.

For this example, suppose the editor offers three candidate selections: the front bird’s wing, the front bird’s body including that wing, and the bird together with its stand. These are hypothetical outputs, not results from a model test.

The first selection might suit an instruction sheet about assembling the wing. The second might suit a product photograph if the stand is excluded from the purchase. The third might be appropriate if the stand is included. None becomes the correct catalogue asset merely because its boundary looks smooth.

Before choosing, the maker needs to decide what the product contains. That decision comes from the catalogue brief and the actual goods. A segmentation result can help implement the decision; it cannot settle an unspecified sales offer.

The partly hidden bird raises another question. Selecting its visible region does not reveal the surface behind the front bird. If a later editing step invents the missing detail, that needs its own review. A clean cut-out and an evidenced view of the complete object are different deliverables.

Several masks do not necessarily mean several items

Now change the task. The same maker wants to count finished ornaments on a workbench. The wing mask and the whole-bird mask overlap because they describe different extents of the same physical item. Counting both as separate products would count one ornament twice.

Removing overlapping selections without a rule would create another problem: two real ornaments can overlap in the photograph. The maker needs a definition of a countable item and a way to handle part–whole relationships, repeated views and obscured items.

A sensible brief for this fictional count might say: count each assembled ornament once; exclude loose components and stands; flag partly obscured items for inspection. It should also state whether unfinished ornaments belong in the total. A stocktake has different requirements from a visual mood board.

This is my practical inference from the distinction between a selected region and an interpreted object. It is not a finding that SAM was tested on this workshop, nor a claim that it can perform an inventory audit unaided.

Video adds continuity to the question

Meta’s SAM 2 research announcement, dated 29 July 2024, extended promptable segmentation to images and videos, using a transformer architecture with streaming memory. Meta’s research project page describes selecting objects and refining predictions with additional prompts.

For the workshop example, following a selected ornament through a clip would be useful for editing its appearance across time. But a video production brief would still need to say which ornament is intended, what to do when it leaves view, and whether a selection should include the stand. Temporal continuity does not write the creative brief.

Meta’s accompanying SA-V dataset page, also dated 29 July 2024, makes another revealing distinction: its mask annotations are class agnostic, without class labels for the videos or masks. A dataset rich in boundaries is not thereby a catalogue of named products.

Write an acceptance rule before making the cut-out

For a small production job, the following questions turn a vague request into something a collaborator can review:

• What is the intended selection: the whole item, a component, or a group?
• Which attachments, supports and surrounding regions should be included?
• What will the result be used for: visual editing, labelling, measurement or counting?
• Which ambiguous or obscured cases need a person to check the original image?
• What will be retained so another editor can understand the choice: source image, selected mask and a short note?

Review the chosen boundary against the actual purpose. In the fictional catalogue job, that includes inspecting the wing gap and the connection to the stand, then checking the cut-out on the intended background. These are proposed production checks, not a validated protocol or a guarantee of accuracy.

There is a related evidence question in MaryChuks.com’s discussion of NeRF view synthesis: what does the generated visual establish, and what must still be checked against the source material?

A strong image workflow gives every stage a defined job. Segmentation supplies a proposed boundary. The brief supplies the meaning of that boundary. Review checks whether the two agree. The outline becomes useful when everyone knows what it represents.

Sources

• Kirillov et al., Segment Anything — ICCV 2023; original preprint released 5 April 2023.
• Ravi et al., SAM 2 research announcement and project overview — announcement dated 29 July 2024.
• Meta, SA-V dataset documentation — dated 29 July 2024.