NEOSTI - a neuromorphic electronic-opto spatial-temporal hybrid image sensor

Why a smarter electronic eye matters

From self-driving cars to household robots, machines increasingly need to see the world clearly and react quickly without draining huge amounts of power. Today’s digital cameras collect sharp images but then must ship massive amounts of data to distant chips or cloud servers for heavy-duty processing, which costs time and energy. This article introduces NEOSTI, a tiny camera system inspired by the human eye that can sense, compress, and understand visual scenes directly on a chip, promising faster, more efficient vision for future devices.

How our eyes inspire new machine vision

The human eye handles enormous streams of visual information with remarkable efficiency. More than 130 million light-sensitive cells in the retina capture incoming light, but smart processing in the eye itself compresses that data by over a hundredfold before sending it along the optic nerve. The retina also handles bright sunlight and dim starlight with ease, while using only a few milliwatts of power. By contrast, most machine vision systems rely on conventional image sensors that simply record pixel brightness and forward raw images to powerful processors, consuming far more energy and struggling to meet split-second response times needed for moving robots, drones, or cars.

A tiny camera that thinks while it looks

NEOSTI (short for Neuromorphic Electronic Opto Spatial Temporal Imager) copies several tricks from biology to tackle these limits. Instead of treating sensing and computing as separate steps, NEOSTI folds them together. First, a patterned optical mask sits in front of the chip and acts like a built-in lens-free filter, performing a kind of sliding selection over the scene before light even reaches the sensor. This trims down the amount of information that must be handled later. Then, specialized pixels convert light into pulses in a nonlinear way that resembles how rods and cones in the eye respond, stretching their useful range from very dark to very bright settings. Finally, a compact neural network built directly on the chip analyzes the resulting patterns to recognize shapes, clothing items, eye positions, or even human actions in short video clips.

Light, pulses, and simple on-chip thinking

Inside NEOSTI, each pixel does more than measure brightness. As light hits a pixel, the voltage drops until it triggers a pulse whose width depends on the light intensity, replacing the usual smooth voltage signal with a timing signal that is easier to handle in digital logic. By slowly shifting the trigger point during each exposure, the pixels mimic the adaptive behavior of the eye, staying sensitive in dim scenes while avoiding overload in bright ones. Directly beneath the sensing area, small processing elements compare neighboring pixels by adding or subtracting their pulse counts, stripping away redundant background and highlighting edges and motion. A hardware pipeline then passes these results through simple steps that echo layers of a neural network, gradually reducing the data while keeping the most informative features.

Putting the system to the test

The researchers evaluated NEOSTI on several well-known image and video tasks. Despite its modest resolution and low power use, the chip correctly classified handwritten digits, fashion items, simple drawings, and synthetic eye images with accuracies above 90 percent in most cases. It also handled a video dataset of people performing actions such as running, jumping, and bending, where it reached about 91 percent accuracy. Tests that turned on only the optical mask or only the on-chip electronics showed that both parts are necessary; using the full system improved accuracy by more than ten percentage points over either part alone. NEOSTI also maintained stable performance as image quality degraded, suggesting that the front-end optical filtering and the on-chip neural network help the system cope with noise and low light.

What this new electronic eye means

For non-specialists, the key message is that NEOSTI shows how future cameras can do much more than capture pictures. By shaping light before it reaches the sensor, turning brightness into robust pulse signals, and adding simple learning circuits directly on the chip, NEOSTI behaves more like a miniature eye and brain combined. It can recognize patterns and actions using far less energy and hardware than traditional setups that rely on large external processors. While further work is needed to scale up resolution and add color, this approach points toward compact, low-power vision modules that could give everyday machines more natural, responsive sight.

Citation: Liu, T., Huang, Z., Wang, X. et al. NEOSTI - a neuromorphic electronic-opto spatial-temporal hybrid image sensor. Nat Commun 17, 4440 (2026). https://doi.org/10.1038/s41467-026-71091-x

Keywords: neuromorphic vision, image sensor, optical computing, edge AI, low power robotics