DriveEmo-FL: in-cabin radar-based emotion sensing for autonomous vehicles smart response

Cars That Can Sense How You Feel

As self-driving cars move closer to everyday reality, a key question remains: how will they know how their passengers are feeling? A ride that is technically safe but leaves people anxious, angry, or carsick will not inspire trust. This paper introduces DriveEmo-FL, a system that lets autonomous vehicles sense passengers’ emotions from subtle body movements—without cameras or wearable gadgets—and then adjust driving and cabin settings in response.

Reading Mood Without Watching Your Face

Many emotion-detection systems today rely on cameras that track facial expressions or wearables that measure heart rate and skin conductance. Inside a car, both approaches have drawbacks: faces can be hidden by sunglasses or poor lighting, and most riders do not want to strap on extra sensors. DriveEmo-FL instead uses millimeter-wave radar, a small device that sends out radio waves and measures how they bounce off a passenger’s upper body. These reflections carry tiny changes in motion that shift the frequency of the returned waves, a pattern known as micro-Doppler. By focusing on shoulders, arms, and head movements, the system can pick up telltale differences between, for example, an excited clap, a fearful shrinking of the shoulders, or the stillness of sadness—all while preserving visual privacy.

From Raw Radar Echoes to Emotional Patterns

The authors build a full pipeline that turns raw radar echoes into emotion estimates in a few dozen milliseconds. First, the signal is cleaned: strong reflections from seats and dashboards are removed, and a mathematical process isolates motion linked to the passenger rather than the moving car. The cleaned data is then transformed into time–frequency images—micro-Doppler "signatures"—that show how motion energy spreads across different speeds over time. In parallel, the system extracts simple motion statistics, such as how abruptly a person’s movement changes (agility), the fastest motion observed (peak velocity), and the total motion energy over a short window. These two streams—rich motion images and compact motion summaries—capture both the shape and intensity of emotional gestures.

A Lightweight Brain for the Car

At the heart of DriveEmo-FL is EmoNet, a compact deep learning model designed to run efficiently on small onboard computers. One branch of EmoNet processes the micro-Doppler images much like a vision network, learning recurring patterns tied to gestures such as clapping, head shaking, or slowly looking around. The other branch processes the three motion statistics, helping distinguish, for example, energetic joy from equally energetic anger. The two streams are fused and fed to shared decision layers that output both the detected activity and the underlying emotion group (happy, sad, angry, or fearful). A final step converts these emotional cues into scores that an autonomous driving system could use to adapt speed, smoothness of maneuvers, lighting, and even the tone of in-car voice assistants.

Learning From Many Cars Without Sharing Secrets

To make the system robust, it must learn from people of different ages, cultures, and body types, riding in different cars and conditions. However, sending everyone’s raw motion data to the cloud would raise privacy concerns. DriveEmo-FL solves this with federated learning: each vehicle trains EmoNet locally using only its own radar data, then shares only encrypted model updates with a central server. The server combines these updates into a stronger global model and sends it back to all cars. In tests with 50 participants performing 12 distinct upper-body gestures, the system reached 94.5% accuracy while running in real time, and it remained dependable in challenging scenarios such as city driving, bright sunlight, low light, and overlapping gestures.

What This Means for Future Rides

Taken together, the results show that self-driving cars can sense and respond to passengers’ emotional states using a small, privacy-friendly radar and an efficient learning model. Without ever recording a face or voice, an autonomous vehicle could notice if a rider seems tense, fearful, or especially cheerful and gently adapt its driving style and cabin environment in response. If refined and combined with other non-intrusive cues, systems like DriveEmo-FL could make future rides feel not just safer on the road, but also more comfortable, understood, and human-centered inside the cabin.

Citation: Imran, N., Alnafisah, K.H., Zhang, J. et al. DriveEmo-FL: in-cabin radar-based emotion sensing for autonomous vehicles smart response. Sci Rep 16, 13600 (2026). https://doi.org/10.1038/s41598-026-43662-x

Keywords: autonomous vehicles, emotion recognition, mmWave radar, in-cabin sensing, federated learning