Sensor fusion of touch & vision in soft manipulators for fruit picking

Gentle robots for fragile fruit

Anyone who has brought home bruised strawberries knows how easily delicate fruit can be damaged. Farmers face the same problem on a massive scale, losing a large share of harvests because fruit is picked at the wrong time or handled too roughly. This study introduces a soft, sensor-packed robot hand designed to see and feel fruit the way a human picker does, so it can harvest crops like strawberries and avocados gently, quickly, and with far less waste.

Why soft hands matter in the field

Many fruits give only subtle clues about ripeness. Some, like citrus, change color slowly and are firm for weeks; others, like berries, soften quickly and can spoil in days. Color alone is often unreliable, and pressing by hand risks bruising. Human pickers have long balanced what they see with what they feel, but growers now struggle with labor shortages and the sheer scale of inspection needed. Existing farm robots mostly rely on rigid metal grippers and cameras placed far from the fruit, which makes it hard to judge color, size, and firmness accurately and can still injure delicate produce.

Seeing and feeling in one compact gripper

The researchers built a star-shaped robot hand with five soft fingers arranged around a central palm. Each finger is made from flexible rubber-like material that bends when air is pumped inside. Hidden within the fingers are clear plastic fibers that guide light; changes in that light reveal how much each finger bends and how hard it is pressing on an object. At the fingertips, U-shaped fibers act as touch sensors, telling the system whether it is holding something soft or firm. In the palm, a tiny camera and a distance sensor sit side by side, giving a close-up view of each fruit and measuring how large it appears. All of this is tied together with small circuit boards and a computer that combine the signals into a single picture of what the gripper is touching and seeing.

How the soft hand performs on fruit

To understand what their device could do, the team first measured the mechanics of the hand. The soft fingers bend up to about two-thirds of a full circle while matching computer simulations, and the gripper can close around objects in under two seconds. It can lift up to one kilogram, more than sixteen times its own weight, and its reach expands from a small central opening to an area large enough to enclose many kinds of fruit. The vision system correctly identified simple shapes, such as cubes and spheres, nearly every time and estimated their size with only a few percent error, even at different viewing distances. By focusing on the hue, or basic color tone, in each pixel, the camera could distinguish reds and greens reliably under fixed lighting, though subtle differences between similar red strawberries remained challenging.

Touch that distinguishes firm from soft

The touch and bend sensors let the robot hand feel fruit in ways that complement its sight. As a finger curls, the light inside its embedded fiber changes in a predictable way, allowing the system to infer how far each finger has bent and how heavy an object might be. When pressed against test samples ranging from very soft rubber to stiff plastic, the fingertip touch sensors showed distinct changes in light intensity that tracked how much force was applied. This allowed the gripper to tell stiff objects from soft ones and to sense the difference between unripe and ripe fruit based on firmness. In demonstrations, the robot gently grasped supermarket strawberries and avocados, measured their color and size, and used the combined data to separate ripe from unripe samples without visible bruising. A built-in twisting module even let the hand rotate around the fruit’s stem to pluck greenhouse strawberries from the plant.

From lab prototype to future farm helper

Together, these abilities show that a single soft gripper can gather rich information about fruit color, shape, size, and firmness while also performing the picking itself. For growers, a tool like this could support more accurate decisions about when to harvest, reduce the need for manual squeezing and sorting, and cut losses from damaged or overripe produce. Although the current system still faces practical issues, such as complex wiring and delicate fibers that can be fouled by fruit juice, it points toward a future in which robots with gentle, sensing hands help bring more intact, high-quality fruit from field to table.

Citation: Mishra, A.K., Ramaswami, A., Shree, V. et al. Sensor fusion of touch & vision in soft manipulators for fruit picking. Nat Commun 17, 4307 (2026). https://doi.org/10.1038/s41467-026-70588-9

Keywords: soft robotics, fruit harvesting, tactile sensing, computer vision, precision agriculture