A studio uses a vision model to measure how long it takes a hand to move a sample from one tray to another. The model finds the right moments in the clip. The report still gives the wrong duration. The script subtracts the two frame numbers and divides by thirty, assuming the video shows a new frame every thirtieth of a second. The video’s own timing says otherwise.
This is a hypothetical example with two separate questions. Did the model recognise the start and finish? And did the software correctly place those observations on a clock? A team can answer the first question perfectly and still get the second wrong. Before comparing movement times, waiting times or events per minute, check how the video analysis turns pictures into time.
A frame number tells you where, not always when
A frame number gives a picture a position in a sequence. A presentation timestamp gives it a position on the media timeline. Apple’s AVFoundation guide to time and media distinguishes presentation time from decode time: the order in which a picture is decoded and the moment it is meant to appear are separate concepts. For a report about what happens on screen, use the presentation timeline.
If a video is genuinely constant frame rate and you have accounted for how frames are numbered and where the timeline starts, a frame count can give the interval. With variable frame spacing, it cannot do that reliably on its own. Imagine four pictures displayed at 0, 33, 67 and 133 milliseconds. The gap from the second to the fourth picture is 100 milliseconds. Two frame steps on a clock assumed to run at exactly thirty frames per second would give about 67 milliseconds. These numbers are invented to show the arithmetic; they are not a measurement of any camera.
An average rate can still describe the whole clip. It just cannot tell you how long every local gap was. That matters when the event you want to measure sits inside one of the uneven sections. A report can be close to right for one clip and misleading for another even when both have the same average rate.
Follow the picture through editing and analysis
First, identify the exact version the model received. Was it the original recording, a trimmed copy or a converted export? A time reported against a trimmed clip may start at the new beginning, while someone reviewing the original needs the corresponding point on that file’s timeline. Tie each observation to the version used and state which timeline the report uses. If people need to switch between the two, keep a mapping and test it against a visible event.
Conversion to a constant rate can be a sensible choice when the downstream tool expects it. But it changes the sequence of pictures that reaches the tool. The FFmpeg documentation explains, for example, that an output frame rate during encoding can be produced by dropping or repeating frames. A repeated frame is not a fresh observation. A dropped frame may have contained the first visible sign of movement. If the model will operate on the converted file, check the conversion with footage that includes the short events you care about.
Sampling for the model introduces another layer. Suppose a tool analyses one frame out of every ten. Save the presentation time of each frame actually analysed. Counting the analysed images will tell you how many checks were performed; it will not, by itself, tell you how long the object was present. If an object appears between checks, the system may only be able to say it appeared after the last clear frame and by the first frame where it was seen. That interval is more useful than a precise-looking time the input cannot support.
Give the report a timing check of its own
Use a short clip with several easily recognised events, one near the start and another well into the recording. Find those events independently in the file the model receives. The FFprobe reference describes how to inspect information at frame level. For an analyst, the useful comparison is between the timestamp attached to the actual frame, the time printed in the report and the position in the video where another person can find it.
- Identify the input. Record the file version, including whether it was trimmed, converted or sampled before analysis.
- Check more than one gap. Include an uneven stretch if it is part of the workload. An opening time that looks right does not verify the rest of the clip.
- Keep the observed times. Store the timestamps alongside model results. If you infer an event boundary between observations, distinguish that estimate from a time you directly observed.
- Repeat the check on the delivered version. If an editor receives a different export, verify that the reported times still point to the events they need.
The related MaryChuks guide to checking audio drift across a whole clip makes a similar point about testing timing beyond the opening seconds. Here, the concern is the time a vision system assigns to an event even when the sound and picture play together perfectly. A dependable report makes the source of that time clear enough for someone to find the moment and reproduce the measurement.
Sources
- Apple: Time and Media Representations (archived AVFoundation Programming Guide).
- FFmpeg: Video Options and FFprobe reference (online project documentation).
Discover more from Marychuks.com AI, Psychology, Business & CreativeVerse
Subscribe to get the latest posts sent to your email.