This Brain Implant Allows People With Paralysis to Speak and Gesture at the Same Time Via a Mind-Controlled Virtual Avatar

The experimental technology is still in the early stage and was fully tested in just two participants. But it could one day help people with motor problems communicate with more nuance or help them control robotic prosthetic limbs

Sara Hashemi | Daily Correspondent

October 6, 2026 12:08 p.m.

Share

Copy linkEmailSMSFacebookXRedditLinkedInBlueskyPrint

Add as preferred source

People with severe paralysis can struggle to communicate with others. While experimental brain-computer interfaces (BCIs) have come a long way in being able to help them convey their thoughts, these technologies have generally been limited to either speech or movement.

 Now, scientists have developed a tool that can do both simultaneously.

The implant, described in the journal Nature Neuroscience on September 14, allows users to control the speech and movement of a virtual avatar with their mind in real time. While the work is still in its early stages, the innovation could one day help people with paralysis communicate with more nuance or serve as a stepping stone to robotic prosthetics that can be controlled with the mind.

“In our everyday life, gestures play an important role in communication,” says study co-author Samantha Brosler, a bioengineer at the University of California, San Francisco, to Carissa Wong at New Scientist. “Saying ‘maybe’ and nodding your head has a very different connotation than if you shake your head.”

Video S3

Watch on

In the new study, Brosler and colleagues focused on two participants, one of whom became paralyzed after a stroke, while the other had the neurodegenerative disease amyotrophic lateral sclerosis (ALS). Each person had an iPhone-size device with 253 electrodes surgically implanted onto their sensorimotor cortex, a brain region involved in processing incoming sensations and outgoing movements. The implant can safely cover such a wide area because it lies on the brain’s surface rather than penetrating it like other BCIs, study co-author Edward Chang, a neurosurgeon at the University of California, San Francisco, tells Jennie Erin Smith at Science.

The team trained computer models to decode the participants’ thoughts as they attempted to say up to ten phrases like “hello” and “nice to meet you” or attempted to perform up to ten gestures such as clapping and waving. They also practiced up to 100 realistic speech-and-gesture combinations. Then, the researchers mapped the brain activity linked to the phrases and gestures to a personalized full-body digital avatar, which could make movements and talk with onscreen text.

During five rounds of a conversation test with the participant with ALS, the decoders got the individual’s intended gesture correct 85 percent of the time, on average, and their intended phrase right 75 percent of the time, on average. The participant who had a stroke had 100 percent accuracy with the speech and gesture decoders during three rounds of a conversation test. The performance disparity might have something to do with the participants’ different causes of paralysis, according to the researchers.

“The authors have taken an important step toward developing highly naturalistic systems to restore function for people with paralysis,” says Daniel Rubin, a neurologist at Massachusetts General Hospital who wasn’t involved in the study, to Science.

While “the results are of very high quality, the level of speech decoding [in this study] is below what has been reported before,” Christian Herff, a computational neuroscientist at Maastricht University in the Netherlands who wasn’t involved in the work, tells Miryam Naddaf at Nature. What’s more, he notes, the BCI decoded only isolated gestures and sentences, whereas real conversations involve continuous streams of speech and movement.

Brosler acknowledges the implant’s current limitations, explaining to the outlet that this study was meant to be a proof of concept. She plans to expand the speech decoder’s vocabulary and give the gesture decoder continuous control over individual body parts, such as moving 15 joints in the hand.

The system could even be a step toward allowing amputees to control prosthetics with their minds.

Communicative gestures “are only one example of a body movement that you may want to do alongside speech,” Brosler tells Science. “I talk while I cook, right? Maybe somebody in the future will want to speak while cooking using their robotic arm.”

Sara Hashemi is a science writer and fact-checker currently based in New York City. Her work has appeared in Sierra, The Body, Maisonneuve magazine and more.