AI Stereo Depth: Check the Match Before You Trust the Distance

Imagine a ceramics studio testing a two-camera setup for making 3D previews. It captures a mug and a bowl on a table, then generates a depth map. The result looks convincing until someone notices that the gap inside the mug handle has been filled in. The software has produced a distance for that part of the picture. The question is whether the two cameras saw enough to justify it.

A stereo depth system has two jobs that are easy to confuse. It must work out which parts of the two pictures correspond, then use the camera geometry to turn that difference into a distance. A problem with either job can produce a tidy image and misleading measurements. For a creator or small business evaluating the technology, the distinction tells you what to check first.

Two views give you a shift to measure

Hold a finger in front of a distant object, then close one eye and swap. The finger appears to move relative to the object. Two cameras looking at the same scene can capture a similar shift. In a rectified stereo pair, the images have been aligned so that corresponding points should sit on the same horizontal row. The difference in their horizontal position is called disparity.

The OpenCV stereo-depth tutorial explains the geometric relationship. In a simplified rectified setup with matching principal points, depth is focal length multiplied by the distance between the cameras, divided by disparity. Focal length and disparity must use compatible image units, such as pixels. If the baseline is measured in centimetres, the resulting depth is in centimetres. The model’s pixel shift is therefore only one ingredient; the camera parameters supply the scale.

Consider a fictional rig where the focal length is 800 pixels and the baseline is 10 centimetres. A disparity of 40 pixels gives a depth of 200 centimetres. A disparity of 20 pixels gives 400 centimetres. An error of one pixel changes the first estimate by about five centimetres and the second by about twenty. These figures illustrate the geometry; they are not a measurement of any particular camera or model. They explain why a near target can look satisfactory while a distant one still needs testing.

What the learning system contributes

In RAFT-Stereo, first posted in September 2021 and presented at 3DV that year, Lahav Lipson, Zachary Teed and Jia Deng describe a learned method that compares features along corresponding image rows and refines an estimate of disparity over successive updates. The paper reports tests on several stereo benchmarks, including Middlebury and ETH3D. The authors’ repository separately describes how to convert disparity to depth using the camera parameters.

Those results concern the systems and datasets the researchers tested. They do not establish that a model will handle a studio’s shiny glaze, plain backdrop or particular lens arrangement. A reflection may appear differently in the two views. Repeated marks can look like several plausible matches. At the edge of the mug, one camera may see background that the other camera cannot see at all. These are different reasons to inspect a result, even if the output gives each spot a colour.

Build a test around the decision you need to make

Decide what a wrong depth would cost before you choose a pass condition. An approximate ordering of foreground and background might be enough for a visual effect. A claim about whether a physical object fits on a shelf requires measured dimensions and a tolerance appropriate to that use. A good result for the visual effect cannot establish that the same setup is suitable for the measurement.

• Start with an object whose position you know. Capture a matte, textured target at several measured distances. Compare the map’s values in the middle of the target before interpreting its edges. A scale error across several clear targets is a reason to check calibration and units.
• Keep the difficult areas in view. Add a shiny mug, a plain surface and an object with a thin gap or edge. Inspect the two input images alongside the output. Mark parts that cannot be checked confidently instead of grading the scene from its best-looking patch.
• Test the capture you plan to use. If the workflow resizes or crops the images, confirm that the calibration used for conversion matches those images. For a moving subject, also check whether the two views represent the same moment.

This is a different task from checking whether an image classifier follows a backdrop, but the same testing principle helps: change one relevant condition at a time. Keep the object still while changing distance. Keep distance fixed while changing the surface. If you alter the rig, lighting and object together, a surprising result will leave you with several explanations to untangle.

If the first test finds a problem, keep the failed pair. Revisit it after you change the lighting, alignment or model, then check a new object with similar features. A fix that works only on the original mug is weaker evidence than one that also survives a second example.

A useful handover includes the original pair, the calibration used for that pair and notes on which areas passed the check. A depth map may be all the viewer sees. The evidence behind it is what lets the next person decide whether to trust a particular distance or treat it as an approximation.

Sources

• Lahav Lipson, Zachary Teed and Jia Deng, RAFT-Stereo: Multilevel Recurrent Field Transforms for Stereo Matching (2021), and the authors’ implementation notes.
• OpenCV, Depth Map from Stereo Images.