Adaptive software framework for context-sensitive, intuitive man-machine-interaction

The aim of project "KiMMI SF" is the development of a software framework for flexible and context-optimized man-machine-interaction. On the one hand, this shall offer online reconfigurable data handling. Furthermore, methods for context-dependent evaluation, selection and weighting of data sources will be developed.

Duration: 01.06.2020 till 31.12.2023
Donee: German Research Center for Artificial Intelligence GmbH & University of Bremen
Sponsor: Federal Ministry for Economic Affairs and Climate Action
German Aerospace Center e.V.
Grant number: DFKI: 50RA2021 Uni: 50RA2022
Application Field: Space Robotics
Related Projects: BesMan
Behaviors for Mobile Manipulation (05.2012- 07.2016)
Dual-arm exoskeleton (01.2011- 12.2013)
Intelligent Man-Machine Interface - Adaptive Brain-reading for assistive robotics (05.2010- 04.2015)
Intelligent Human-Robot Collaboration (03.2015- 06.2016)
Intelligent Structures for Mobile Robots (05.2010- 08.2013)
Learning Intelligent Motions for Kinematically Complex Robots for Exploration in Space (05.2012- 04.2016)
Technologies and Human-Robot Collaboration for Surface EVA Exploration Activities and Training in European Analogue Environments (09.2013- 08.2016)
Recupera REHA
Full-body exoskeleton for upper body robotic assistance (09.2014- 12.2017)
Semi-autonomous cooperative exploration of planetary surfaces including the installation of a logistic chain as well as consideration of the terrestrial applicability of individual aspects (05.2013- 12.2017)
AI-based Qualification of Deliberative Behaviour for a Robotic Construction Kit (08.2018- 07.2021)
Flexible Interaction for infrastructures establishment by means of teleoperation and direct collaboration; transfer into industry 4.0 (07.2017- 12.2021)
Models, methods and tools for the model based software development of robots (06.2015- 05.2018)
Related Robots: HunterSE
Z1-Arm with gripper
Full Body Exoskeleton
Exoskeleton for upper body robotic assistance
Related Software: Bagel
Biologically inspired Graph-Based Language
Signal Processing and Classification Environment written in Python
Reconfigurable Signal Processing and Classification Environment
Behavior Optimization and Learning for Robots
Robot Construction Kit

Project details

KiMMI SF is a joint research project with the aim of realizing a software framework for flexible human-machine interaction which is optimal for the respective context.

Specifically, in subproject 1 “Development of an adaptive software framework for MMI”, led by DFKI RIC, an adaptive software framework is being developed that enables online reconfigurable data processing based on different data sources. The framework shall enable a multimodal evaluation of the situation of the human being, the agent and the environment.

In subproject 2 “Development of methods for online context recognition”, led by the University of Bremen, methods for context-dependent evaluation, selection and weighting of data soures will be developed to enable online interpretation of context information based on multimodal data and to derive the human condition. The software framework developed in the project is a prerequisite for context-dependent intuitive human-machine interaction, which is to be made possible through collaboration in the project. This will allow the interaction between humans and robotic agents to continue flexibly, i.e. even when the context changes.

Due to the adaptive nature of the software framework to be developed in KiMMI SF for flexible human-machine interaction that is optimal for the respective context, it is expected, among other things, that downtimes due to otherwise unrecognized context and intention changes in humans can be minimized. Thus, it can serve as an essential contribution for the success of future missions. KiMMI SF builds on existing technology developments in previous projects in the field of space travel, particularly in the area of providing services (PS).


RH5 Manus: Robot Dance Generation based on Music Analysis Driven Trajectory Optimization

Musical dancing is an ubiquitous phenomenon in human society. Providing robots the ability to dance has the potential to make the human robot co-existence more acceptable. Hence, dancing robots have generated a considerable research interest in the recent years. In this paper, we present a novel formalization of robot dancing as planning and control of optimally timed actions based on beat timings and additional features extracted from the music. We showcase the use of this formulation in three different variations: with input of a human expert choreography, imitation of a predefined choreography, and automated generation of a novel choreography. Our method has been validated on four different musical pieces, both in simulation and on a real robot, using the upper-body humanoid robot RH5 Manus.

RH5 Manus: Background of robot dance generation based on music analysis driven trajectory optimization



Investigations into Exploiting the Full Capabilities of a Series-Parallel Hybrid Humanoid using Whole Body Trajectory Optimization
Melya Boukheddimi, Rohit Kumar, Shivesh Kumar, Carpentier Justin, Frank Kirchner
In 2023 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), (IROS-2023), 01.10.-05.10.2023, Detroit, Michigan, IEEE, Oct/2023.
Context-attentive robot reconfiguration for collaborative human-machine space missions
Isabelle Kien, Manuel Meder, Proneet Kumar Sharma
In 13th EASN International Conference, (EASN-2023), 05.9.-09.9.2023, Salerno, Italy, IOP Journal of Physics, Sep/2023.
Detection and recognition of human manipulation building blocks
Lisa Gutzeit
Mar/2023. Universität Bremen.
The VVAD-LRS3 Dataset for Visual Voice Activity Detection
Adrian Lubitz., Matias Valdenegro-Toro., Frank Kirchner.
In Proceedings of the 18th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications - HUCAPP, (VISIGRAPP-2023), 19.2.-21.2.2023, Lissabon, SciTePress, pages 39-46, 2023. INSTICC. ISBN: 978-989-758-634-7.
A Reference Implementation for Knowledge Assisted Robot Development for Planetary and Orbital Robotics
Mehmed Yüksel, Thomas M. Röhr, Marko Jankovic, Wiebke Brinkmann, Frank Kirchner
In Acta Astronautica, Elsevier Ltd., volume 2023, pages 1-16, 2023.
CoBaIR: A Python Library for Context-Based Intention Recognition in Human-Robot-Interaction
Adrian Auer, Lisa Gutzeit, Frank Kirchner
In 2023 32nd IEEE International Conference on Robot and Human Interactive Communication (RO-MAN), (RO-MAN-2023), 28.8.-31.8.2023, Busan, IEEE, 2023.
Fast Dynamic Walking with RH5 Humanoid Robot
Ivan Bergonzani, Mihaela Popescu, Shivesh Kumar, Frank Kirchner
In 2023 IEEE-RAS 22nd International Conference on Humanoid Robots (Humanoids), (Humanoids), 12.12.-14.12.2023, Austin, TX, IEEE, 2023. ISBN: 979-8-3503-0327-8.


Unsupervised Segmentation of Human Manipulation Movements into Building Blocks
Lisa Gutzeit, Frank Kirchner
In IEEE Access, IEEE, volume 10, pages 125723-125734, Dec/2022.
Ontology-Driven Robot Design for Future Orbital and Planetary Robotics with korcut
Mehmed Yüksel, Thomas M. Röhr
European Aeronautics Science Network International Conference (EASN), Oct/2022.
Hierarchical Segmentation of Human Manipulation Movements
Lisa Gutzeit
In Proc. of the 26th International Conference on Pattern Recognition, (ICPR-2022), 21.8.-25.8.2022, Montreal, QC, IEEE Computer Society, pages 2742-2748, Aug/2022.
A Holistic Approach for Enhancing Data Integrity and Reliability in Human-Robot Interaction
Manuel Meder, Kashmira Shinde, Dennis Hemker, Sadique Siddiqui, Teena Hassan, Nina Hoyer, Elsa Andrea Kirchner
In Proceedings of SpaceCHI 2.0: Human-Computer Interaction for Space Exploration, (SpaceCHI 2.0-2022), 01.5.-01.5.2022, New Orleans, LA, ACM, May/2022.
A Bayesian Approach to Context-based Recognition of Human Intention for Context-Adaptive Robot Assistance in Space Missions
Adrian Auer, Octavio Arriaga, Teena Hassan, Nina Hoyer, Elsa Andrea Kirchner
In Proceedings of SpaceCHI 2.0 - Human-Computer Interaction for Space Exploration - A Workshop at CHI 2022, (SpaceCHI 2.0-2022), 01 .5.-01.5.2022, New Orleans, LA, ACM, May/2022.
Multi-modal manipulation planning for an upper-torso humanoid system
Sankaranarayanan Natarajan, Frank Kirchner
In 2022 IEEE-RAS 21st International Conference on Humanoid Robots, (Humanoids-2022), 28.11.-30.11.2022, Ginowan City, Okinawa, IEEE, 2022.

Back to the list of projects
last updated 04.01.2024
to top