An attention-based multimodal deep learning framework integrating EEG and ECG for enhanced stress detection

Why tracking stress objectively matters

Stress shapes how we think, feel, and stay healthy, yet most of us only notice it once it starts to cause problems. This study presents a new way to track stress in real time using signals from the brain and the heart together, rather than asking people how they feel or looking at one signal at a time. The goal is to turn invisible strain into clear, reliable information that could one day support health care, workplace wellbeing, and personal mental health tools.

Limits of simply asking people how they feel

Today, stress is often measured with questionnaires that ask people to recall how tense they felt over the past days or weeks. These self-reports can be skewed by memory, culture, or the wish to appear “fine,” and they cannot capture fast changes in the body during a stressful moment. Meanwhile, wearable devices now collect detailed biological data, offering a way to read stress signals directly from the body. The challenge is to interpret these rich streams of data in a way that reflects both mind and body, rather than treating stress as a single number.

What brainwaves and heartbeats reveal

The researchers focus on two types of signals. Electroencephalogram, or EEG, tracks brainwaves and can show shifts linked with focus, worry, and emotional strain. Electrocardiogram, or ECG, records heartbeats and provides heart rate variability, a subtle measure of how flexibly the heart responds to demands. Under stress, brain rhythms change and heart rate variability usually drops, but not always in the same way for every person or situation. By combining these two views, the study aims to paint a richer picture of how stress shows up across the nervous system, from thoughts and feelings down to the heart.

A smart system that blends brain and heart signals

The team designed a deep learning framework that treats EEG and ECG as two complementary channels. First, they carefully clean and reshape the raw signals. Brain activity is filtered, cut into short time windows, and turned into images that show how power in different brainwave bands changes over time. Heart data goes through its own pipeline, capturing both the raw rhythm of beats and classic variability measures. Three powerful image-recognition networks, originally trained on large picture datasets, are adapted to extract patterns from these transformed signals. An attention-based fusion layer then learns how to weigh contributions from brain and heart, and even from each network, depending on which is most informative in a given moment.

Testing the system across different kinds of stress

To see how well this approach works, the authors trained and tested it on two independent public datasets. One, called WESAD, includes brain and heart recordings from volunteers who moved through neutral, tense social tasks, and amusing activities. The other, called CASE, adds more subjects and a mix of mental challenges and emotional scenes, offering a tougher test of general use. Across both sets, the combined EEG-plus-ECG system could distinguish neutral, stressed, and positive states with about 95.7 percent accuracy. This clearly surpassed versions that relied on brain data alone, heart data alone, or simpler ways of merging signals, and it held up reasonably well even when trained on one dataset and tested on the other.

From lab results to everyday stress support

Beyond high accuracy, the model is built with practical use in mind. It can run fast enough for near real-time monitoring and can be simplified for devices with limited computing power. The attention weights also offer a window into which features matter most, highlighting stress-related patterns like increased frontal brain activity and reduced heart rate variability. In plain terms, the study shows that listening to both the brain and the heart, and letting an intelligent system decide how to blend their voices, gives a more faithful reading of stress. While more work is needed to bring this into clinics, workplaces, and consumer gadgets, it marks a concrete step toward objective, non-intrusive tools that help people understand and manage their mental load.

Citation: Kumar, R., Krishnan, S.B., Yadav, R.K. et al. An attention-based multimodal deep learning framework integrating EEG and ECG for enhanced stress detection. Sci Rep 16, 15188 (2026). https://doi.org/10.1038/s41598-026-44499-0

Keywords: stress detection, EEG, ECG, multimodal deep learning, physiological signals