LINC

CLINICAL APPLICATION OF LINC FOR THE CONTROL OF EXTERNAL DEVICES BASED ON IMPLANTED ELECTRODES

Summary

The main focus of the project was on machine learning for neural control signals in the context of “LiNC-concept-based” BCIs (brain-computer interfaces with shared control mediated by the user’s higher-order, cognitive brain signal and an autonomous intelligence of the prosthesis). Such BCI devices lie at the heart of the long-term BrainLinks-BrainTools (BLBT) vision. They can be used, e.g., by paralyzed patients for real-time control of an assistive robotic device, or to aid speech restoration. Extraction of reliable neural control signals is, however, a challenging task. Several aspects must be considered, such as where to position the recording electrodes, which neuronal signal features and behavioral (e.g., movement, speech) parameters to control, or how to adjust the data processing. In our project, thanks to the BLBT cluster, we were able to successfully address these highly interdisciplinary topics, spanning over multiple scientific fields, such as clinical neurology, experimental and theoretical neuroscience, machine learning, robotics, and linguistics. Among other achievements, we have created highly accurate deep learning models for movement decoding and developed several visualization methods to better understand what features the networks extract from the brain signal -- a crucial requirement to use the networks in a medical context and a promising way to gain further scientific insights on the underlying decoding problems. To advance deep learning methodology in general and for these applications in particular, we have introduced several improvements of the training process, improving the state of the art on several standard deep learning image processing benchmarks and also in EEG decoding. Furthermore, we have provided a detailed spatio-temporal description of cortical activity underlying speech perception, speech production and of orofacial non-speech behaviors during real-life conversations. The majority of our studies has focussed on natural, real-life situations, as opposed to (behaviorally rather artificial) experimental paradigms, very often used in animal models. This research not only allows addressing scientific questions without animal subjects but it also allows unique insights into neuronal activity of behaving individuals.

Publications and Achievements

1. Burget*, F., Fiederer*, L. D. J., Kuhner*, D., Völker*, M., Aldinger*, J., Schirrmeister, R. T., Do, C., Boedecker, J., Nebel, B., Ball, T., and Burgard, W. (2017) Acting Thoughts: Towards a Mobile Robotic Service Assistant for Users with Limited Communication Skills. 2017 European Conference on Mobile Robots (ECMR), 1-6.
2. Derix, J., Iljina (Glanz), O., Schulze-Bonhage, A., Aertsen, A., and Ball, T. (2012) "Doctor" or "darling?" Decoding the communication partner from ECoG of the anterior temporal lobe during non-experimental, real-life social interaction. Frontiers in Human Neuroscience 6:251.
3. Derix, J., Iljina (Glanz), O., Weiske, J., Schulze-Bonhage, A., Aertsen, A., and Ball, T. (2014a) From speech to thought: The neuronal basis of cognitive units in non-experimental, real-life communication investigated using ECoG. Frontiers in Human Neuroscience 8:383.https://www.frontiersin.org/articles/10.3389/fnhum.2014.00383/full
4. Derix, J., Yang, S., Lüsebrink, F., Fiederer, L.D., Schulze-Bonhage, A., Aertsen A., Speck, O., and Ball, T. (2014b) Visualization of the amygdalo-hippocampal border and its structural variability by 7T and 3T magnetic resonance imaging. Human Brain Mapping 35:4316-29.                       
5. Fiederer, L. D. J., Lahr, J., Vorwerk, F., Lucka, M., Aertsen, A., Wolters, C. H., Schulze-Bonhage, A., and Ball, T. (2016) Electrical Stimulation of the Human Cerebral Cortex by Extracranial Muscle Activity: Effect Quantification With Intracranial EEG and FEM Simulations. IEEE Transactions on Biomedical Engineering 63, 12:2552-63.
6. Fiederer, L. D. J., Vorwerk, J., Lucka, F., Dannhauer, M., Yang, S., Dümpelmann, M., Schulze-Bonhage, A., Aertsen, A., Speck, O., Wolters, C. H., and Ball, T. (2016) The Role of Blood Vessels in High-Resolution Volume Conductor Head Modeling of EEG. NeuroImage 128:193-208.
7. Glanz (Iljina), O., Derix, J., Kaur, R., Schulze-Bonhage, A., Auer, P., Aertsen, A., and Ball, T. (2018) Real-life speech production and perception have a shared premotor-cortical substrate. Scientific Reports 8(1):8898.
8. Hammer, J., Fischer, J., Ruescher, J., Schulze-Bonhage, A., Aertsen, A., and Ball, T. (2013) The role of ECoG magnitude and phase in decoding position, velocity, and acceleration during continuous motor behavior. Frontiers in Neuroscience 7:200.https://www.nature.com/articles/s41598-018-26801-x
9. Hammer, J., Pistohl, T., Fischer, J., Kršek, P., Tomášek, M., Marusič, P., Schulze-Bonhage, A., Aertsen, A., and Ball, T. (2016) Predominance of Movement Speed Over Direction in Neuronal Population Signals of Motor Cortex: Intracranial EEG Data and A Simple Explanatory Model. Cerebral Cortex 26:2863-81.
10. Hammer, J., Schirrmeister, R.T., Hartmann, K., Marusic, P., Krsek, P., Schulze-Bonhage, A., Hutter, F., and Ball, T. (to be submitted) Simultaneous representation of amplitude and phase EEG features by deep convolutional neural networks.
11. Hammer, J., Tomasek, M., Marusic, P., Krsek, P., Schulze-Bonhage, A., Ball, T. (to be submitted) Temporal encoding properties of kinematic parameters during continuous hand movements in intracranial EEG.
12. Hammer, J., Tomasek, M., Marusic, P., Krsek, P., Schulze-Bonhage, A., Ball, T. (to be submitted) Distribution changes of decodable information in low frequencies of intracranial EEG during slow and fast hand movements.
13. Heilmeyer, F. A., Schirrmeister, R. T., Fiederer, L. D. J., Völker, M., Behncke, J., and Ball, T. (2018) A framework for large-scale evaluation of deep learning for EEG. arXiv preprint arXiv:1806.0774111
14. Iljina (Glanz), O., Derix, J., Schirrmeister, R. T., Schulze-Bonhage, A., Auer, P., Aertsen, A., and Ball, T. (2017) Neurolinguistic and machine-learning perspectives on direct speech BCIs for restoration of naturalistic communication. Brain Computer Interfaces 4(3):186-199.
15. Kern, M., Bert, S., Glanz (Iljina), O., Schulze-Bonhage, A., and Ball, T. (2019) Smiling, laughing, speech production: Sparse and action-specific activation of the human motor cortex during non-experimental, real-life orofacial movements, Communications Biology
16. Kern, M., Aertsen, A., Schulze-Bonhage, A., and Ball, T. (2013) Heart cycle-related effects on event-related potentials, spectral power changes, and connectivity patterns in the human ECoG, NeuroImage 81:178-90.
17. Kuhner*, D., Fiederer*, L. D. J.,Aldinger*, J., Burget*, F., Völker*, M., Schirrmeister, R. T., Do, C., Boedecker, J., Nebel, B., Ball, T., and Burgard, W. (2018) Deep Learning Based BCI Control of a Robotic Service Assistant Using Intelligent Goal Formulation. BioRxiv 282848.https://www.frontiersin.org/articles/10.3389/fnhum.2012.00251/full
18. Ruescher, J., Iljina (Glanz), O., Altenmüller, D. M., Aertsen, A., Schulze-Bonhage, A., and Ball, T. (2013) Somatotopic mapping of natural upper- and lower-extremity movements and speech production with high gamma electrocorticography. NeuroImage 81:164-177.
19. Schröer, S., Killmann, I., Frank, B., Völker, M., Fiederer, L., Ball, T., and Burgard, W. (2015) An autonomous robotic assistant for drinking. Proceedings of the IEEE International Conference on Robotics and Automation (ICRA'15) 6482-6487.
20. Thinnes-Elker, F.*, Iljina (Glanz), O.*, Apostolides, J. K., Kraemer, F., Schulze-Bonhage, A., Aertsen, A., and Ball, T. (2012) Intention concepts and brain-machine interfacing. Frontiers in Psychology 3:455. * equal contributions.
21. Feurer, M., Klein, A., Eggensperger, K., Springenberg, J., Blum, M. and Hutter, F. (2015) Efficient and Robust Automated Machine Learning. Advances in Neural Information Processing Systems 28 (NIPS 2015)
22. Mendoza, H. and Klein, A. and Feurer, M. and Springenberg, J. and Hutter, F. (2016) Towards Automatically-Tuned Neural Networks International Conference on Machine Learning 2016, AutoML Workshop
23. Loshchilov, I. and Hutter, F. (2016) Online Batch Selection for Faster Training of Neural Networks. International Conference on Learning Representations (ICLR 2016, Workshop Track)
24. Loshchilov, I. and Hutter, F. (2016) CMA-ES for Hyperparameter Optimization of Deep Neural Networks. International Conference on Learning Representations (ICLR 2016, Workshop Track)
25. Schubert, T. and Eggensperger, K. and Gkogkidis, A. and Hutter, F. and Ball, T. and Burgard, W. (2016) Automatic Bone Parameter Estimation for Skeleton Tracking in Optical Motion Capture. IEEE International Conference on Robotics and Automation (ICRA 2016)
26. Meinel, A. and Eggensperger, K. and Tangermann, M. and Hutter, F. (2016) Hyperparameter Optimization for Machine Learning Problems in BCI (Abstract). Proceedings of the International Brain Computer Interface Meeting 2016
27. Schirrmeister, R. and Springenberg, T. and Fiederer, L. and Glasstetter, M. and Eggensperger, K. and Tangermann, M. and Hutter, F. and Burgard, W. and Ball, T. (2017) Deep learning with convolutional neural networks for EEG decoding and visualization. Human Brain Mapping 38 (2017): 5391--5420
28. Loshchilov, I. and Hutter, F. SGDR: Stochastic Gradient Descent with Warm Restarts. International Conference on Learning Representations (ICLR 2017)
29. Falkner, Stefan and Klein, Aaron and Hutter, Frank (2018) BOHB: Robust and Efficient Hyperparameter Optimization at Scale. 35th International Conference on Machine Learning (ICML 2018)
30. Loshchilov, I. and Hutter, F. Fixing Weight Decay Regularization in Adam. (2018) Under review at: International Conference on Learning Representations (ICLR 2018)
31. Behncke, J., Schirrmeister, R. T., Burgard, W., & Ball, T. (2018, January). The signature of robot action success in EEG signals of a human observer: Decoding and visualization using deep convolutional neural networks. In Brain-Computer Interface (BCI), 2018 6th International Conference on (pp. 1-6). IEEE.
32. Behncke J., Schirrmeister R. T., Völker M., Hammer J., Marusič P., Schulze-Bonhage A., Burgard W., and Ball T. (2018) Cross-paradigm pretraining of convolutional networks improves intracranial EEG decoding. IEEE International Conference on Systems, Man, and Cybernetics 2018
33. Hartmann, K. G., Schirrmeister, R. T., & Ball, T. (2018, January). Hierarchical internal representation of spectral features in deep convolutional networks trained for EEG decoding. In Brain-Computer Interface (BCI), 2018 6th International Conference on (pp. 1-6). IEEE.
34. Heilmeyer F.A., Schirrmeister R.T., Fiederer L.D.J., Völker M., Behncke J., Ball T. (2018) A framework for large-scale evaluation of deep learning for EEG. IEEE International Conference on Systems, Man, and Cybernetics 2018 https://arxiv.org/abs/1806.07741
35. Schirrmeister, R.T., Gemein, L., Eggensperger, K., Hutter, F., & Ball, T. (2017, December) Deep learning with convolutional neural networks for decoding and visualization of EEG pathology. In Signal Processing in Medicine and Biology Symposium (SPMB), 2017 IEEE (pp. 1-7). IEEE.
36. Schirrmeister, R.T, Chrabąszcz, P., Hutter, F. & Ball, T. (2018) Training Generative Reversible Networks In ICML 2018 workshop on Theoretical Foundations and Applications of Deep Generative Models
37. Völker, M., Schirrmeister, R. T., Fiederer, L. D., Burgard, W., & Ball, T. (2018, January). Deep transfer learning for error decoding from non-invasive EEG. In Brain-Computer Interface (BCI), 2018 6th International Conference on (pp. 1-6). IEEE.
38. Kellmeyer P, Grosse-Wentrup M, Schulze-Bonhage A, Ziemann U, Ball T. Electrophysiological correlates of neurodegeneration in motor and non-motor brain regions in amyotrophic lateral sclerosis-implications for brain-computer interfacing. J Neural Eng. 2018 Aug; 15(4):041003. doi: 10.1088/1741-2552/aabfa5.
39. Kellmeyer P., Mueller O., Feingold-Polak R. and Levy-Tzedek S. Social robots in rehabilitation: A question of trust. Science Robotics. 2018 Aug 15 3(21). doi.org/scirobotics.aat1587