CLINICAL APPLICATION OF LINC FOR THE CONTROL OF EXTERNAL DEVICES BASED ON IMPLANTED ELECTRODES
Relevant for Research Area
The main focus of the project was on machine learning for neural control signals in the context of “LiNC-concept-based” BCIs (brain-computer interfaces with shared control mediated by the user’s higher-order, cognitive brain signal and an autonomous intelligence of the prosthesis). Such BCI devices lie at the heart of the long-term BrainLinks-BrainTools (BLBT) vision. They can be used, e.g., by paralyzed patients for real-time control of an assistive robotic device, or to aid speech restoration. Extraction of reliable neural control signals is, however, a challenging task. Several aspects must be considered, such as where to position the recording electrodes, which neuronal signal features and behavioral (e.g., movement, speech) parameters to control, or how to adjust the data processing. In our project, thanks to the BLBT cluster, we were able to successfully address these highly interdisciplinary topics, spanning over multiple scientific fields, such as clinical neurology, experimental and theoretical neuroscience, machine learning, robotics, and linguistics. Among other achievements, we have created highly accurate deep learning models for movement decoding and developed several visualization methods to better understand what features the networks extract from the brain signal -- a crucial requirement to use the networks in a medical context and a promising way to gain further scientific insights on the underlying decoding problems. To advance deep learning methodology in general and for these applications in particular, we have introduced several improvements of the training process, improving the state of the art on several standard deep learning image processing benchmarks and also in EEG decoding. Furthermore, we have provided a detailed spatio-temporal description of cortical activity underlying speech perception, speech production and of orofacial non-speech behaviors during real-life conversations. The majority of our studies has focussed on natural, real-life situations, as opposed to (behaviorally rather artificial) experimental paradigms, very often used in animal models. This research not only allows addressing scientific questions without animal subjects but it also allows unique insights into neuronal activity of behaving individuals.