BCI enables avatar to speak and gesture simultaneously

Researchers at the University of California, San Francisco (UCSF) have developed a brain-computer interface (BCI) that enables people with severe paralysis to control speech and gestures simultaneously through a digital avatar. Previous BCI systems have supported either verbal or non-verbal communication, but according to the researchers, this is the first system capable of enabling both at the same time.

Using machine learning, the team analyzed brain activity generated when participants attempted to speak and make physical gestures simultaneously. In two participants, these signals were successfully translated into speech and movements of a full-body virtual avatar. The findings are published in Nature Neuroscience.

Communication beyond words

People with conditions such as amyotrophic lateral sclerosis (ALS) or brainstem stroke can develop severe paralysis that limits or eliminates both verbal and non-verbal communication. Eye-tracking technology can enable patients to select text that is subsequently converted into speech, but the process can be slow, offers limited opportunities for expression and may be physically exhausting.

The UCSF researchers have therefore been working on technology designed to make BCI-mediated communication more natural. In previous research, they implanted a thin strip of electrodes, known as an electrocorticography (ECoG) array, on the motor cortex. Computer algorithms then translated recorded brain signals into commands controlling a digital representation of the head and face.

For the new study, the researchers extended this approach to a full-body avatar. Participants with varying levels of vocal tract and bodily paralysis attempted to speak specific phrases and perform common gestures, such as waving or giving a thumbs-up. They performed these actions both separately and simultaneously.

Combining speech and gestures

The study showed that combining speech and gestures is neurologically more complex than simply adding together two separate sets of brain signals. Although there was some overlap, brain activity during simultaneous speech and gestures differed substantially from the patterns recorded when either action was performed separately.

This distinction proved important when training the algorithms used to interpret brain activity. Decoders trained on data collected during simultaneous speech and movement were more successful at recognizing combined expressions than models trained only on isolated speech and gesture data.

In two participants, the BCI translated these signals into commands that allowed a virtual avatar to produce both verbal and physical expressions. The proof-of-concept therefore demonstrates that a BCI can potentially control multiple layers of communication at the same time.

Towards a wireless BCI

According to study leader Edward Chang, human conversation involves much more than spoken words. Gestures and other physical expressions also play an important role in communication. The findings suggest that BCIs could eventually restore some of this broader expressive capacity for people who can no longer communicate conventionally because of paralysis.

The experimental system currently relies on wired connections between implanted sensors and external equipment that processes the brain signals. This limits its practical use outside a research environment.

The UCSF team plans to test a fully implantable, wireless version of the system as a next step. Such a device could offer better prospects for long-term use. However, the current research remains a proof-of-concept and does not yet establish that the technology is ready for routine clinical or everyday use.

BCI for speech

Last year, researchers at Stanford University developed a BCI capable of decoding inner speech in real time. The technology could eventually offer a less physically demanding communication method for people with severe paralysis, including patients with ALS or brainstem injuries.

Four participants with complete paralysis had microelectrodes implanted in their motor cortex. An AI model was trained on neural signals recorded while they attempted to speak words and while they imagined saying them. Although inner speech produced weaker brain activity, the patterns were sufficiently distinct to decode imagined sentences from a 125,000-word vocabulary with 74 percent accuracy in a proof-of-concept.

The researchers also developed a security mechanism that activates decoding using a chosen “thought password.” The experimental phrase was recognized with more than 98 percent reliability. The system could also distinguish between inner speech and attempted speech, potentially allowing users greater control over what is communicated.