A high-performance neuroprosthesis for speech decoding and avatar control using ECoG – Kaylo Littlejohn

Register now

Talk Description:

Speech neuroprostheses have the potential to restore communication to people living with paralysis, but naturalistic speed and expressivity are elusive. Here we use high-density surface recordings of the speech cortex in a clinical-trial participant with severe limb and vocal paralysis to achieve high-performance real-time decoding across three complementary speech-related output modalities: text, speech audio and facial-avatar animation. We trained and evaluated deep-learning models using neural data collected as the participant attempted to silently speak sentences. For text, we demonstrate accurate and rapid large-vocabulary decoding with a median rate of 78 words per minute and median word error rate of 25%. For speech audio, we demonstrate intelligible and rapid speech synthesis and personalization to the participant’s pre-injury voice. For facial-avatar animation, we demonstrate the control of virtual orofacial movements for speech and non-speech communicative gestures. The decoders reached high performance with less than two weeks of training. Our findings introduce a multimodal speech-neuroprosthetic approach that has substantial promise to restore full, embodied communication to people living with severe paralysis.

About the speaker:

Kaylo Littlejohn is an EECS Ph.D. student at UC Berkeley advised by Professor Gopala K. Anumanchipalli and Dr. Edward F. Chang. His research is focused on restoring lost function to those who have severe paralysis via the use of speech and avatar neuroprostheses. Kaylo’s research enabled the first demonstration of intelligible speech-synthesis from neural activity in a person who has lost the ability to communicate.

You might also like

February 27, 2025

Live

Starts: 10:00 am MT

Ends: 11:00 am MT

 A Snapshot of Innovation: Creating Tools to Advance Human Subchronic Research

Speaker: Join Anthony Khong, an associate product manager at Blackrock Neurotech, for insights on how we work to bring cutting-edge, innovative […]

November 14, 2024

Virtual Webinar

Starts: 10:00 am MDT

Ends: 11:00 am MDT

Pushing the Limits of ICMS for Tactile Feedback

Speaker: Charles Greenspon, a Research Assistant Professor at the University of Chicago, uses intracortical microstimulation (ICMS) to restore the sense […]

April 4, 2024

Virtual

Starts: 11:00 pm MT

Ends: MT

Cortical representation of learning social interactions in freely moving rhesus monkeys

Speaker: Dr. Melissa Franch, Postdoctoral Associate at Baylor College of Medicine Bio: Growing up with a brother with profound autism, […]

December 1, 2023

Live and On-Demand

Starts: 10:00 am MT

Ends: 11:00 am MT

Grasp and speech Brain-Machine Interfaces from the posterior parietal cortex

Talk abstract: The ability to communicate with our surroundings and interacting with it through grasping are invaluable to human independence. […]

June 7, 2023

On Demand

Starts: 10:00 am MDT

Ends: 11:00 am MDT

Building the Most Advanced Brain-Controlled Neuroprosthetics

Listen in while Dr. John Downey, a Staff Scientist from the University of Chicago provides an overview of the state-of-the-art […]

May 9, 2023

On Demand Event

Starts: 9:00 am PDT

Ends: 10:00 am PDT

Capturing and Enhancing Episodic Memories Made in the Wild

One of the ultimate goals of neuroscience is to explain real-world, human behavior in terms of the activities of the […]

May 9, 2023

On-Demand

Starts: 11:00 am

Ends: 12:00 pm

Who are the BCI Pioneers and Why?

Do you want to know what it’s like to use an implanted BCI device? Come hear the first-hand experiences and […]

May 9, 2023

On Demand Event

Starts: 12:00 pm EDT

Ends: 1:00 pm EDT

Uncovering How the Human Brain Responds to Direct Electrical Stimulation

Listen to this webinar presented by Angelique C Paulk, Instructor, Department of Neurology, Massachusetts General Hospital, and Harvard Medical School. […]

May 9, 2023

On-Demand

Starts: 12:00 pm MDT

Ends: 1:00 am MDT

Conscious Visual Perception and Restoring it in Blindness

Listen to Pieter R. Roelfsema, Netherlands Institute for Neuroscience, Amsterdam; Institute de la Vision, Paris to find out how future […]