NeuroBots

BRAIN-CONTROLLED INTELLIGENT ROBOTIC DEVICES


Relevant for Research Area

B - Core Technologies

C - Applications


Summary

Brain-controlled assistive robots hold the promise of restoring autonomy to paralyzed patients. Existing approaches are based on low-level, continuous control of robotic devices, resulting in a high cognitive load for their users. In the NeuroBots project, in contrast, we enhance prosthetic devices with a certain degree of autonomy and adaptivity to enable control on a higher cognitive level. To achieve this, we develop new methods and technologies in core areas of brain-machine interfaces, as well as artificial intelligence. This includes innovative approaches to brain-signal decoding with deep neural networks, efficient motion planning and improved perception for mobile robots and manipulators, novel methods for deep reinforcement learning, hierarchical planning with user feedback, and evaluation of formal methods for safety guarantees. The different components are continually integrated in an architecture based on the Robot Operating System (ROS), realizing a demonstrator of the BrainLinks-BrainTools LiNC concept.


Research Status

The most outstanding result of the project is a fully integrated system that realizes the BrainLinks-BrainTools LiNC concept, combining state-of-the-art online decoding of neuronal control signals with deep neural networks, high-level hierarchical planning with graphical user interface based on the planner’s world knowledge, as well as novel perception and low-level robot planning algorithms for mobile robots. Improved high-level brain-signal decoding and closed-loop human-robot interaction are in the focus of current research.

Project Publications

J. Aldinger, R. Mattmüller, and M. Göbelbecker. “Complexity Issues of Interval Relaxed Numeric Planning”. In: 38th German Conference on Artificial Intelligence (KI 2015). 2015.

J. Aldinger and B. Nebel. “Interval Based Relaxation Heuristics for Numeric Planning with Action Costs”. In: Proceedings of the Tenth International Symposium on Combinatorial Search (SoCS 2017). 2017.

J. Aldinger and B. Nebel. “Interval Based Relaxation Heuristics for Numeric Planning with Action Costs”. en. In: Proceedings of the 40th German Conference on Artificial Intelligence (KI 2017). Vol. 10505. Cham: Springer International Publishing, 2017, pp. 15–28. isbn:
978-    3-319-67189-5 978-3-319-67190-1. doi: 10.1007/978-3-319-67190-1_2.

A. Amiranashvili, A. Dosovitskiy, V. Koltun, and T. Brox. “Motion Perception in Rein-forcement Learning with Dynamic Objects”. en. In: Conference on Robot Learning (CoRL). 2018.

A. Amiranashvili, A. Dosovitskiy, V. Koltun, and T. Brox. “TD or not TD: Analyzing the Role of Temporal Di˙erencing in Deep Reinforcement Learning”. en. In: International Conference on Learning Representations (ICLR). 2018.

J. Behncke, R. T. Schirrmeister, W. Burgard, and T. Ball. “The signature of robot ac-tion success in EEG signals of a human observer: Decoding and visualization using deep convolutional neural networks”. In: 2018 6th International Conference on Brain-Computer Interface (BCI). Jan. 2018, pp. 1–6. doi: 10.1109/IWW-BCI.2018.8311531.

J. Behncke, R. T. Schirrmeister, M. Völker, A. Schulze-Bonhage, W. Burgard, and T. Ball. “Cross-paradigm pretraining of convolutional networks improves intracranial EEG decoding”. In: 2018 IEEE International Conference on Systems, Man, and Cybernetics (SMC). arXiv: 1806.09532. June 2018.

J. Behncke, R. T. Schirrmeister, W. Burgard, and T. Ball. “The Role of Robot Design in Decoding Error-related Information from EEG Signals of a Human Observer”. In: 6th In-ternational Congress on Neurotechnology, Electronics and Informatics. Oct. 2018, pp. 61–
66.    isbn: 978-989-758-326-1.

J. Boedecker, J. T. Springenberg, J. Wülfing, and M. Riedmiller. “Approximate real-time optimal control based on sparse gaussian process models”. In: 2014 IEEE Symposium on Adaptive Dynamic Programming and Reinforcement Learning (ADPRL). IEEE. 2014, pp. 1–8.

F. Burget and M. Bennewitz. “Stance Selection for Humanoid Grasping Tasks by In-verse Reachability Maps”. In: IEEE International Conference on Robotics and Automation (ICRA). Washington, Seattle, USA, 2015.

F. Burget, M. Bennewitz, and W. Burgard. “BI2RRT ∗: An Eÿcient Sampling-Based Path Planning Framework for Task-Constrained Mobile Manipulation”. In: IEEE/RSJ Interna-tional Conference on Intelligent Robots and Systems (IROS). Daejeon, South Korea, 2016.

F. Burget, A. Hornung, and M. Bennewitz. “Whole-Body Motion Planning for Manipula-tion of Articulated Objects”. In: IEEE International Conference on Robotics and Automa-tion (ICRA). Karlsruhe, Germany, 2013.

F. Burget, L. D. J. Fiederer, D. Kuhner, M. Völker, J. Aldinger, R. T. Schirrmeister, C. Do, J. Boedecker, B. Nebel, T. Ball, et al. “Acting thoughts: Towards a mobile robotic service assistant for users with limited communication skills”. In: European Conference on Mobile Robots (ECMR). IEEE. 2017, pp. 1–6.

C. Do and W. Burgard. “Accurate Pouring with an Autonomous Robot Using an RGB-D Camera”. In: The 15th International Conference on Intelligent Autonomous Systems (IAS). Baden Baden, Germany, 2018.

C. Do, C. Gordillo, and W. Burgard. “Learning to Pour using Deep Deterministic Pol-icy Gradients”. In: Proceedings of the International Conference on Intelligent Robots and Systems (IROS). Madrid, Spain, 2018.

C. Do, T. Schubert, and W. Burgard. “A Probabilistic Approach to Liquid Level Detection in Cups Using an RGB-D Camera”. In: IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). Daejeon, South Korea, 2016.

M. Göbelbecker. “Assisting With Goal Formulation for Domain Independent Planning”. In: 38th German Conference on Artificial Intelligence (KI 2015). 2015.

K. G. Hartmann, R. T. Schirrmeister, and T. Ball. “Hierarchical internal representation of spectral features in deep convolutional networks trained for EEG decoding”. In: 2018 6th International Conference on Brain-Computer Interface (BCI). Jan. 2018, pp. 1–6. doi: 10.1109/IWW-BCI.2018.8311493.

H. Hatefi, R. Wimmer, B. Braitling, L. M. F. Fioriti, B. Becker, and H. Hermanns. “Cost vs. time in stochastic games and Markov automata”. en. In: Formal Aspects of Computing 29.4 (July 2017), pp. 629–649. issn: 1433-299X. doi: 10.1007/s00165-016-0411-1.

F. A. Heilmeyer, R. T. Schirrmeister, L. D. J. Fiederer, M. Völker, J. Behncke, and T. Ball. “A framework for large-scale evaluation of deep learning for EEG”. In: 2018 IEEE International Conference on Systems, Man, and Cybernetics (SMC). arXiv: 1806.07741. June 2018.

G. Kalweit and J. Boedecker. “Uncertainty-driven Imagination for Continuous Deep Rein-forcement Learning”. In: Proceedings of the 1st Annual Conference on Robot Learning. Ed. by S. Levine, V. Vanhoucke, and K. Goldberg. Vol. 78. Proceedings of Machine Learning Research. PMLR, 2017, pp. 195–206.

D. Kuhner, J. Aldinger, F. Burget, M. Göbelbecker, W. Burgard, and B. Nebel. “Closed-Loop Robot Task Planning Based on Referring Expressions”. In: Proc. of the IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). Madrid, Spain, 2018.

T. Lampe and M. Riedmiller. “Approximate model-assisted neural fitted Q-iteration”. In: 2014 International Joint Conference on Neural Networks (IJCNN). IEEE. 2014, pp. 2698–2704.

T. Lampe, L. D. Fiederer, M. Voelker, A. Knorr, M. Riedmiller, and T. Ball. “A Brain-Computer Interface for High-Level Remote Control of an Autonomous, Reinforcement-Learning-Based Robotic System for Reaching and Grasping”. In: Proceedings of the 19th International Conference on Intelligent User Interfaces. IUI ’14. New York, NY, USA: ACM, 2014, pp. 83–88. isbn: 978-1-4503-2184-6. doi: 10.1145/2557500.2557533.

D. Maier, C. Lutz, and M. Bennewitz. “Integrated Perception, Mapping, and Footstep Planning for Humanoid Navigation Among 3D Obstacles”. In: IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). Tokyo, Japan, 2013.

K. Scheibler, F. Neubauer, A. Mahdi, M. Fränzle, T. Teige, T. Bienmüller, D. Fehrer, and B. Becker. “Accurate ICP-based Floating-Point Reasoning”. In: Formal Methods in Computer-Aided Design, FMCAD 2016. IEEE, 2016.

K. Scheibler, L. Winterer, R. Wimmer, and B. Becker. “Towards Verification of Arti-ficial Neural Networks”. In: Proceedings of the 18th Workshop “Methoden und Beschrei-bungssprachen zur Modellierung und Verifikation von Schaltungen und Systemen” (MBMV). Ed. by U. Heinkel, M. Rößler, and D. Kriesten. Chemnitz, Germany: Technische Univer-sität Chemnitz, Germany, Mar. 2015, pp. 30–40. isbn: 978-3-944640-34-1.

R. Schirrmeister, P. Chrab¡szcz, F. Hutter, and T. Ball. “Training Generative Reversible Networks”. In: ICML 2018 workshop on Theoretical Foundations and Applications of Deep Generative Models. arXiv: 1806.01610. July 2018.

R. T. Schirrmeister, J. T. Springenberg, L. D. J. Fiederer, M. Glasstetter, K. Eggensperger,
M.    Tangermann, F. Hutter, W. Burgard, and T. Ball. “Deep learning with convolutional neural networks for EEG decoding and visualization”. en. In: Human Brain Mapping 38.11 (Nov. 2017), pp. 5391–5420. issn: 1097-0193. doi: 10.1002/hbm.23730.

S. Schröer, I. Killmann, B. Frank, M. Völker, L. Fiederer, T. Ball, and W. Burgard. “An autonomous robotic assistant for drinking”. In: IEEE International Conference on Robotics and Automation (ICRA). 2015, pp. 6482–6487.

J. T. Springenberg, A. Dosovitskiy, T. Brox, and M. Riedmiller. “Striving for Simplicity: The All Convolutional Net”. In: arXiv:1412.6806, also appeared at ICLR 2015 Workshop Track. 2015.

J. T. Springenberg and M. Riedmiller. “Improving Deep Neural Networks with Proba-bilistic Maxout Units”. In: arXiv:1312.6116, also appeared at ICLR 2014 Workshop Track. 2014.

M. Völker, R. T. Schirrmeister, L. D. J. Fiederer, W. Burgard, and T. Ball. “Deep transfer learning for error decoding from non-invasive EEG”. In: 2018 6th International Conference on Brain-Computer Interface (BCI). Jan. 2018, pp. 1–6. doi: 10.1109/IWW-BCI.2018.8311491.

M. Völker, J. Hammer, R. T. Schirrmeister, J. Behncke, L. D. J. Fiederer, A. Schulze-Bonhage, P. Marusi£, W. Burgard, and T. Ball. “Intracranial Error Detection via Deep Learning”. In: 2018 IEEE International Conference on Systems, Man, and Cybernetics (SMC). arXiv: 1805.01667. May 2018.

M. Völker, L. D. J. Fiederer, S. Berberich, J. Hammer, J. Behncke, P. Kršek, M. Tomášek,
P.    Marusi£, P. C. Reinacher, V. A. Coenen, M. Helias, A. Schulze-Bonhage, W. Burgard, and T. Ball. “The dynamics of error processing in the human brain as reflected by high-gamma activity in noninvasive and intracranial EEG”. In: NeuroImage 173 (June 2018), pp. 564–579. issn: 1053-8119. doi: 10.1016/j.neuroimage.2018.01.059.

M. Watter, J. Springenberg, J. Boedecker, and M. Riedmiller. “Embed to Control: A Locally Linear Latent Dynamics Model for Control from Raw Images”. In: Advances in Neural Information Processing Systems 28. Ed. by C. Cortes, N. Lawrence, D. Lee, M. Sugiyama, R. Garnett, and R. Garnett. 2015, pp. 2728–2736.

D. Welke, J. Behncke, M. Hader, R. T. Schirrmeister, A. Schönau, B. Eßmann, O. Müller,
W.    Burgard, and T. Ball. “Brain Responses During Robot-Error Observation”. In: Kogni-tive Systeme. Vol. 1. Sept. 2017. doi: 10.17185/duepublico/44533.

L. Winterer, S. Junges, R. Wimmer, N. Jansen, U. Topcu, J. Katoen, and B. Becker. “Motion planning under partial observability using game-based abstraction”. In: 2017 IEEE 56th Annual Conference on Decision and Control (CDC). Dec. 2017, pp. 2201–2208. doi: 10.1109/CDC.2017.8263971.

Neurobots Demonstrator: https://www.youtube.com/watch?v=Ccor_RNHUAA