CoTracker3: Did the Video AI See What Happened Next?

A dot follows a moving object, disappears behind a hand, then returns to the right place. The demonstration looks convincing. But before calling it evidence that an AI system can follow events live, ask a question the animation cannot answer: which frames had the model already received?

A finished recording contains information that was unavailable when the action happened. Using that information can be entirely appropriate for editing footage. It changes what the result establishes about a system expected to respond as events unfold. CoTracker3 offers a useful, concrete way to understand this distinction.

What CoTracker3 actually tracks

The official CoTracker repository describes a model that follows selected points through video. A user can choose points or sample them on a grid. The example code returns predicted positions and predicted visibility. These are outputs about where a queried point travels and whether it is visible, rather than a ready-made explanation of the scene.

The CoTracker3 preprint, submitted on 15 October 2024, introduced a revised tracker and training approach using automatically generated labels for real videos. This is an established research example, not a newly announced product. The important feature for this discussion is that the work provides online and offline variants.

For a general reader, think of placing a tiny marker on the corner of a box in the opening frame. The task is to estimate where that same point belongs later. Whether the box is being packed correctly, whether its contents are damaged, or why somebody moved it are separate questions.

Offline means a different information budget

In the paper’s method, the online variant processes overlapping windows and moves forwards through the video. The offline variant tracks in both temporal directions. A completed clip can therefore support a different kind of analysis from an arriving stream. Here, “offline” describes video processing; it does not simply mean that the computer has no internet connection.

Nor should “online” be read as a promise of an instantaneous answer for every incoming frame. The paper describes windowed processing. For a live application, the relevant practical question is how much footage must arrive before an output is available. The label alone does not specify the delay your particular workflow can tolerate.

Consider a hypothetical craft tutorial. A maker’s hand covers a marked corner of a wooden piece, then moves away. An editor preparing the finished tutorial may reasonably use later footage when deciding where to place a graphic. A system meant to guide the maker during that covered interval has a different information constraint. Both tasks are legitimate; they need different demonstrations.

A smooth trail does not explain the hidden interval

The researchers’ project page includes demonstrations of tracking through occlusion, meaning periods when a point is hidden. Those examples are useful evidence of the behaviour shown on those clips. They do not establish that every hidden movement in a new recording will be recovered correctly.

An estimated location behind a hand deserves a different reading from a clearly visible corner. The practical implication is to preserve that distinction when presenting results. If your graphic draws an uninterrupted line, explain whether parts of it represent estimates during hidden periods. Otherwise the presentation can imply a continuous observation that the camera never supplied.

This is related to the distinction explored in MaryChuks.com’s guide to points, identity and behavioural claims in animal tracking. Here the extra question concerns timing: what information could support the estimate at the moment you intend to use it?

Build a demonstration around the decision deadline

If you are assessing a proposed workflow, start by writing a plain sentence: “We need the position by this moment, using footage available by that moment.” This is an editorial recommendation for evaluating the use case, not a benchmark reported by the researchers.

For a retrospective video edit, the deadline might be delivery of the finished file. For an on-screen cue during filming, it might be shortly after each movement. Do not borrow the first task’s polished output as the sole test of the second.

A useful trial would keep the clip, query points and evaluation rule fixed while changing how footage is supplied. Record the model variant, the last frame available for each output, when that output becomes available, and whether later processing revises it. Inspect difficult intervals separately rather than judging only the attractive opening seconds.

Include a point passing behind an obstruction, a fast movement and an ordinary visible section. Keep a human-reviewed reference for the visible positions you assess. For hidden intervals, state how any reference position was established; do not create supposed ground truth merely by tracing what another model guessed.

Choose the processing mode for the job

For an independent creator, the useful outcome may be a stable graphic added after recording. In that hypothetical use, access to the whole clip is a resource to use openly. There is no reason to reject retrospective processing simply because it benefits from later evidence.

For a business owner considering an interactive service, the meaningful trial is closer to the proposed operating conditions. Request an output log that shows availability times as well as positions. Ask what happens when the point becomes hidden and how the interface communicates uncertainty. A demonstration that never exposes those moments leaves an important part of the proposal untested.

The lesson is to match the claim to the information supplied. A tracker can be valuable for reconstructing a recorded sequence without the same demonstration proving timely performance on a stream. Before judging how well AI followed the action, establish when it was allowed to see the ending.

Sources

Karaev and colleagues, CoTracker3 preprint (2024), especially the online-model method; official repository, including separate processing examples and output fields; official project demonstrations. The workflow examples and assessment recommendations above are editorial proposals, not reported experiments.