Spatiotemporal dynamics and substates underlie emotional signalling in facial movements

Why our faces in motion matter

Everyday life is full of quick glances, raised eyebrows, and half smiles that steer our social worlds. Yet most research has treated facial expressions like still photographs. This study asks a more realistic question: how do the moving patterns of our faces over time carry emotional meaning, both when we are silent and when we are speaking?

A few core movement patterns

The researchers recorded the faces of 43 volunteers as they expressed happiness, sadness, and anger in two ways: with silent expressions and while saying a neutral sentence in an emotional tone. Using automated facial analysis, they tracked how groups of muscles around the eyes, cheeks, and mouth changed over hundreds of tiny time steps. They then used mathematical tools to condense this rich motion into a smaller set of basic patterns that repeatedly showed up across people and emotions.

Upper face, lower face, and mixed signals

For silent expressions, the facial dynamics could be explained by just three main components: one dominated by movements in the upper face, one by the lower face, and one that linked lower and upper movements. Different emotions emerged from different blends of these components over time. Anger involved strong, coordinated changes in both upper and lower regions. Happiness relied most heavily on lower face actions like smiling. Sadness used more moderate shifts across all three patterns. When the team trained a computer classifier on these dynamic signatures, it could tell which emotion was being expressed from the motion alone with high accuracy.

Layering emotion onto speech

When people spoke while expressing emotion, the same basic idea held, but the patterns mixed differently. The three components now tended to combine mouth movements needed for speech with eyebrow and eye changes that signal feeling. The classifier still performed well, though not as perfectly as for silent expressions, reflecting the extra complexity of juggling both speaking and emotional display. This suggests that our faces reuse a small set of motion building blocks, flexibly reshaping them to fit both verbal and emotional needs at once.

Hidden mini-phases inside each expression

The team then zoomed in further on how expressions unfold second by second. By clustering the motion data, they found three recurring "substates" that cut across all expressions: relaxed periods with little movement, fast transition periods as the face shifts into or out of an expression, and sustain periods where the expression is held. These mini-phases differed in speed and complexity depending on the emotion and on whether the person was speaking. For example, transitions were especially fast and distinctive for happy expressions, and the overall sequence of substates was more structured in silent expressions than during speech.

Matching human perception

To test whether these low-dimensional patterns actually matter to observers, the researchers created simplified animations showing only moving dots on the face, removing all other cues. Forty five new volunteers watched these animations and judged which emotion they saw. Human choices lined up closely with the model’s predictions, and the same small set of motion patterns was enough to predict people’s emotion labels at rates well above chance. This indicates that both senders and receivers rely on a compact set of facial dynamics when giving and reading emotional cues.

What this means for everyday interaction

Overall, the study suggests that despite the apparent richness of our facial behavior, emotional signalling depends on just a few core movement patterns and a handful of brief phases as expressions rise, hold, and fade. This streamlined structure may help the brain control many facial muscles efficiently while still conveying clear feelings, even during conversation. The findings offer a blueprint for modelling natural emotional expressions in fields like social robotics and may also help explain why some clinical conditions involve subtle but important differences in facial communication.

Citation: Cuve, H.C.J., Sowden-Carvalho, S. & Cook, J.L. Spatiotemporal dynamics and substates underlie emotional signalling in facial movements. Sci Rep 16, 15686 (2026). https://doi.org/10.1038/s41598-026-46726-0

Keywords: facial expressions, emotion recognition, facial movement dynamics, nonverbal communication, emotive speech