Sentinel for confidence-aware multi-object tracking

Keeping Track of Many Things at Once

From self-driving cars and delivery robots to smart security cameras and sports broadcasts, modern machines increasingly need to follow many people or objects at the same time. Yet real life is messy: crowds block the view, cameras blur, and detectors are unsure whether a faint shape is a person or just background. This paper presents “Sentinel,” a new way for computers to track many moving objects more reliably by explicitly reasoning about uncertainty—how sure or unsure the system is about what it sees.

Why Tracking in the Real World Is Hard

Multi-object tracking systems usually work in two steps. First, they detect objects in each video frame. Second, they connect these detections over time to form continuous paths, or trajectories, for each individual. Existing systems tend to trust only the most confident detections, throwing away weaker ones to avoid false alarms. That helps precision but hurts recall: during motion blur or partial blockage, many real people are seen only weakly and get dropped. At the same time, traditional trackers often delete a trajectory after it has been missing for a fixed number of frames. This age-based rule fails in real crowds, where someone may vanish behind others for a while and then reappear, causing their track to be cut into pieces and their identity to be reassigned.

A Tracker That Knows When It Is Sure or Unsure

Sentinel tackles both problems by treating each trajectory as having its own evolving level of confidence. One part of the system, called Confidence Aware Association, looks at how often a track has been successfully matched, how often it has lately failed, and how strong its recent detections were. Based on this history, it classifies each track as confident, uncertain, or at risk. For confident tracks, whose motion is well predicted, Sentinel leans heavily on where the person is expected to be, and it pays less attention to visual appearance. This helps avoid mixing up people who look alike but stand in different places. For risky tracks, which may have just come out of occlusion or have shaky predictions, the system does the opposite: it widens the search area and relies more on how the person looks than on where the simple motion model says they should be.

Giving Disappearing Tracks a Second Chance

The second component, called the Survival Boosting Mechanism, steps in when a track is in danger of disappearing. Instead of immediately deleting a track after a fixed number of missing frames, Sentinel keeps a “survival score” that grows as the track stays unmatched. As the risk grows, the system actively searches among low-confidence detections—signals the detector is unsure about—to find plausible candidates that might be the same person. It gently adjusts how much it trusts position, appearance, and physical movement limits, gradually allowing for more positional error while demanding consistent appearance and realistic motion. When a weak but plausible detection passes these tests, Sentinel temporarily boosts its internal confidence so it can compete with stronger detections in the main matching step, giving the original track a chance to continue instead of being replaced.

Putting Sentinel to the Test

The authors tested Sentinel on three demanding benchmark collections. MOT17 covers varied street scenes with pedestrians, MOT20 focuses on extremely crowded situations with heavy occlusion, and DanceTrack follows dancers who move in nonlinear, unpredictable ways while often wearing similar outfits. Across these datasets, Sentinel consistently improved measures that emphasize keeping each person’s identity intact over time, such as the Identification F1-score and the Higher Order Tracking Accuracy. It also reduced the number of identity switches and track fragments compared with well-known trackers that either treat all detections the same or terminate tracks passively. Although Sentinel introduces some extra computation and can create a few more false positives when it leans on weak detections, it remains fast enough for real-time use in most scenarios.

What This Means for Everyday Technology

In plain terms, Sentinel makes machine vision systems more patient and more thoughtful. Instead of dropping people as soon as they are hard to see or blindly trusting every blurry hint, it continuously asks how sure it is about each track and adjusts its behavior accordingly. That strategy pays off in the most challenging settings: busy sidewalks, dense crowds, or fast-moving performers. The work suggests that future tracking systems—whether in cars, drones, or cameras—will be more reliable if they treat uncertainty as a first-class signal, using it to decide when to be cautious, when to search harder, and when to give a nearly lost object one more chance to stay in view.

Citation: Yang, HS., Park, SW., Sim, CB. et al. Sentinel for confidence-aware multi-object tracking. Sci Rep 16, 13571 (2026). https://doi.org/10.1038/s41598-026-43938-2

Keywords: multi-object tracking, computer vision, object detection, occlusion handling, trajectory continuity