Imagine a creator reviewing a video of someone examining an illustrated journal. The viewer’s eyebrows draw together. An AI summary describes the reaction as dissatisfaction. The creator considers changing the cover, although the person might have been concentrating on small lettering or reacting to something outside the frame.
This hypothetical example contains a small but consequential leap: a visible movement has become an explanation of someone’s inner experience. Before that explanation shapes a creative or business decision, ask what the camera recorded, what the system inferred and what the person actually said.
Three claims that deserve separate treatment
Describing a movement, assigning an emotion label and establishing someone’s feelings are different tasks. “Their eyebrows moved downwards” concerns visible behaviour. “They looked annoyed” adds an interpretation. “They disliked the cover” adds both an emotional judgement and a proposed cause.
Lisa Feldman Barrett and colleagues examined this distinction in a review published on 17 July 2019. They found substantial variation in how people communicate emotions across situations, cultures and individuals. Particular movements sometimes accompany familiar emotions more often than chance, yet similar movements can accompany different emotions or non-emotional states. The review challenges a simple mapping from a facial configuration to a specific feeling; it does not claim that faces carry no useful social information.
For the journal creator, a description of the visible moment is a useful starting point. It leaves room to investigate why the moment happened.
What changes when the frame widens?
A PNAS study published on 12 March 2025, led by Maya Lecker, tested human perception using videos from intense fear-inducing situations. Across 12 preregistered experiments involving 4,180 participants, observers saw faces alone, context without faces or the full videos. Isolated faces did not communicate fear reliably; the context-containing versions were much more clearly perceived as fearful.
The experiments measured human observers’ judgements. Evaluating an AI system’s performance would require a separate test. The researchers also lacked direct measurements of the filmed people’s feelings and physiology during the events, so mixed emotions could not be excluded. The result concerns how observers interpreted those recordings. It should not become a claim that adding background scenery gives a machine certain access to someone’s emotional experience.
Ask who supplied the comparison label
The EMOTIC research by Ronak Kosti and colleagues offers a helpful technical example. The authors describe images annotated for apparent emotions, using crowd workers who saw people in context. Their model combined information about the person with information from the wider scene. The study also examined agreement between annotators.
Those annotations record how other people interpreted an image. They are not statements made by the photographed individuals about their own feelings. Predicting those labels well and confirming a person’s inner state are therefore separate achievements. Such research can help investigate how apparent emotions are perceived. Its results need to be described in terms of the labels actually studied.
When considering a tool, ask what its evaluation calls a correct answer. Was the comparison supplied by an observer, an actor following instructions or the person describing their own experience? Each answers a different question. A headline accuracy figure needs that explanation before it can guide a decision.
Review a creator’s video in three layers
Return to the fictional journal review. A practical approach is to keep three kinds of note:
- Recorded behaviour: the person pauses at the cover and brings the journal closer.
- Possible interpretation: the lettering may be difficult to read, or the person may be studying the illustration.
- Participant account: what the person says when invited to explain that moment.
A useful follow-up would be: “What were you looking at when you brought the journal closer?” Asking “Why did the cover annoy you?” builds the AI’s interpretation into the question. The first wording gives the participant more room to describe their own experience.
Suppose, within this hypothetical example, the participant explains that the subtitle was difficult to read. The creator now has a specific issue to examine: whether that lettering needs changing. A note that simply said “negative reaction” would have offered much less direction.
Their explanation still belongs alongside the observed task and other feedback. One person’s account does not establish how every customer will respond. Our guide to customer interviews develops this practice of keeping evidence, interpretation and open questions distinct.
Give AI a task you can inspect
If you use AI during a review, set a narrow brief: identify candidate moments for a person to revisit, describe visible changes and mark uncertain interpretations. Check whether your chosen tool can actually do that before relying on its output.
Keep the original recording available for comparison. If the tool mentions a smile, pause or gesture, revisit that passage yourself. If it supplies an emotion score, ask what that number means and how it was validated. Avoid copying a score into a customer profile without an explanation of what it measures.
Keep a time reference and the object under discussion with each note. If later editing removes the surrounding scene, the note should still identify the passage it describes. Before revising a design, write down the specific change you intend to test and why the evidence supports it. That makes the proposed revision easier to assess.
Let the person’s meaning remain open
For creators and business owners, the useful question is often concrete: which words were unclear, which step caused difficulty or which design detail deserved another look? Those questions can lead to changes you can test.
A facial label may suggest a moment worth discussing. Treat that as an invitation to ask, listen and compare evidence. Leave space for a person to say that your interpretation missed something. Understanding the reaction should improve the work and the conversation, rather than end the conversation with a label.
Sources
- Barrett and colleagues: facial movements and emotion inference (2019), with the November 2019 figure corrigendum.
- Lecker and colleagues: context and perceived fear (2025); author-hosted full paper.
- Kosti and colleagues: EMOTIC paper, author version posted 30 March 2020; university-hosted manuscript.
Discover more from Marychuks.com AI, Psychology, Business & CreativeVerse
Subscribe to get the latest posts sent to your email.
One thought on “AI Emotion Recognition: Keep the Context in View”