Dynamic hand exercise recognition for game-based finger rehabilitation

Why video games can help tired hands heal

For many people recovering from stroke, arthritis, or other arm injuries, doctors prescribe frequent hand and finger exercises. These moves are simple but must be repeated over and over, which can quickly become boring. This study explores how to turn those clinical finger exercises into the controls of a video game, so that patients practice the same movements while playing and having fun.

Turning therapy moves into game controls

The researchers focused on seven common finger exercises used in clinics, such as making a fist, spreading the fingers, touching the thumb to each finger, and stretching the thumb. Instead of using generic hand signals like counting on fingers, they built a new image collection based entirely on these true therapy moves. Fourteen volunteers were photographed performing each exercise with both hands under different lighting and backgrounds, creating 2,800 color images that better reflect what real patients would do during finger rehabilitation.

Teaching a computer to read the hand

To recognize the exercises automatically, the team used a well known image recognition network called VGG16. Rather than training a huge model from scratch, they used transfer learning, where knowledge from a system already trained on millions of everyday pictures is adapted to a new task. First, they kept the original network fixed and added their own final layers to tell the seven exercises apart. Then they allowed the last few layers of the base network to adjust slightly to the new hand images. This fine tuning step helped the system pick up the subtle differences between similar finger positions.

How well the system understood the hand

On new images from people the network had never seen during training, the model correctly recognized the exercises about 82 percent of the time before fine tuning and about 85 percent after fine tuning. The largest mistakes happened when telling apart middle finger and ring finger exercises, especially when images were flipped left to right, because the patterns of bent and straight fingers looked very similar. By using special heat maps, the team showed that the model focused on regions where these fingers and the thumb appear, revealing why some poses were often confused. Even so, the overall accuracy was high enough for smooth use in a game setting.

Playing bowling with your fingers

To test the system in practice, the researchers connected it to a simple bowling video game. A camera captured a live view of the player’s hand, the recognition model identified the current finger exercise, and each gesture was mapped to one of the game’s actions, such as moving the player up or down or throwing the ball. In this way, every move in the game required a real rehabilitation exercise, so that playing meant practicing. Fifteen healthy volunteers tried the system in two short sessions, after a brief practice period to learn which gesture triggered which action.

What players felt during the game

After playing, participants filled out standard questionnaires that measure enjoyment, motivation, sense of control, immersion, and whether the game caused pain. Most players reported that the game was enjoyable, felt that they were getting better at using the gestures, and did not feel pressured or tense. They also said the game was not too hard to control and generally pain free, suggesting that the exercises were comfortable. Although not everyone said they would play every day, many found the idea effective and interesting for supporting hand exercise.

Why this matters for future rehab games

This work shows that using a recognition system built around real therapy exercises can make game based finger training both practical and engaging. Even though the model sometimes misread individual frames, the game responded to a continuous stream of hand positions, so brief errors rarely affected play. The authors see their dataset and results as a starting point for more advanced systems and for future studies with older adults and people with hand impairments, where such exergames could help keep repetitive rehabilitation both accurate and enjoyable.

Citation: Ajani, O.S., Darlan, D., Aboyeji, E. et al. Dynamic hand exercise recognition for game-based finger rehabilitation. Sci Rep 16, 15007 (2026). https://doi.org/10.1038/s41598-026-42693-8

Keywords: finger rehabilitation, hand gesture recognition, exergames, transfer learning, virtual therapy