Learning and Verifying Complex Behaviours for Humanoid Robots

The validation of systems based on deep learning for use in safety-critical applications proves to be inherently difficult, since their sub symbolic mode of operation does not provide adequate levels of abstraction for representation and proof of correctness. The VeryHuman project aims to synthesize such levels of abstraction by observing and analysing the behaviour of upright walking of a two-legged humanoid robot. The theory to be developed is the starting point for the definition of an appropriate reward function to optimally control the movements of the humanoid by means of enhanced learning, as well as for verifiable abstraction of the corresponding kinematic models, which can be used to validate the behaviour of the robot more easily.

Duration: 01.06.2020 till 31.05.2024
Donee: German Research Center for Artificial Intelligence GmbH
Sponsor: Federal Ministry of Education and Research
Grant number: 01IW20004
Partner: Cyber-Physical Systems (CPS) Research Department, German Research Center for Artificial Intelligence (DFKI)
Application Field: Assistance- and Rehabilitation Systems
Logistics, Production and Consumer
SAR- & Security Robotics
Space Robotics
Related Projects: D-Rock
Models, methods and tools for the model based software development of robots (06.2015- 05.2018)
Flexible Interaction for infrastructures establishment by means of teleoperation and direct collaboration; transfer into industry 4.0 (07.2017- 12.2021)
AI-based Qualification of Deliberative Behaviour for a Robotic Construction Kit (08.2018- 07.2021)
Related Robots: RH5
Humanoid robot as an assistance system in a human-optimized environment
RH5 Manus
Humanoid robot as an assistance system in a human-optimized environment
Related Software: HyRoDyn
Hybrid Robot Dynamics
Machina Arte Robotum Simulans
Node Level Data Link Communication
Robot Construction Kit

Project details

Overall workflow in the project VeryHuman. Source: DFKI, Foto: Daniel Harnack
Biologically inspired control algorithms for robots have been proven very successful. Often, these algorithms use techniques such as reinforcement learning or optimal control to perform sophisticated movement patterns with a robot (for e.g. humanoid walking). However, two main challenges exist for these learning-based approaches:
  • A robust hardware of the robot along with an accurate simulation of the system is required. For example, the robot can be subjected to a large amount of holonomic constraints including internal closed loops and external contacts which pose challenges to the accuracy of the simulation.
  • Second, control algorithms of this kind can be hard to implement due to lack of knowledge of reward and constraints. As an example, consider the upright walking movement for a two-legged humanoid robot. It is not immediately clear how one can specify the task of “upright walking”. We might try to relate different body parts (head above shoulders, shoulders above waist, waist above legs), use physical stability criteria (centre of pressure, zero moment point etc), but do these really specify walking and what are non-trivial properties? This leads to the non-trivial task of defining a suitable reward function for (deep) reinforcement learning approaches or cost function for optimal control approaches along with constraints.
This project aims at three basic research questions:
  • How can we formulate and prove properties of a complex humanoid robot, and
  • how can we efficiently combine reinforcement learning and optimal control-based approaches, and
  • how can we make use of symbolic properties to derive a reward function in a deep reinforcement learning approach or optimal control approach for complex use cases such as humanoid walking.
These three research questions are closely interwoven and are dealt with in three work areas that lead to the overarching goal of the project: a methodology to develop a hybrid of deep (reinforcement) learning and optimization based control of a robot together with a corresponding rational reconstruction of its observed and future behavior. This reconstruction is based on observations of the robot’s movements and general knowledge of physics (rigid body dynamics). The overall approach is shown in Fig 2. The demonstration scenario includes showing walking for a complex series-parallel humanoid robot RH5 which has been recently developed at DFKI-RIC (see Fig. 1).


RicMonk: A Three-Link Brachiation Robot with Passive Grippers for Energy-Efficient Brachiation

This paper presents the design, analysis, and performance evaluation of RicMonk, a novel three-link brachiation robot equipped with passive hook-shaped grippers. Brachiation, an agile and energy-efficient mode of locomotion observed in primates, has inspired the development of RicMonk to explore versatile locomotion and maneuvers on ladder-like structures. The robot's anatomical resemblance to gibbons and the integration of a tail mechanism for energy injection contribute to its unique capabilities. The paper discusses the use of the Direct Collocation methodology for optimizing trajectories for the robot's dynamic behaviors and stabilization of these trajectories using a Time-varying Linear Quadratic Regulator. With RicMonk we demonstrate bidirectional brachiation, and provide comparative analysis with its predecessor, AcroMonk - a two-link brachiation robot, to demonstrate that the presence of a passive tail helps improve energy efficiency. The system design, controllers, and software implementation are publicly available on GitHub.

M-RoCK+VeryHuman: Whole-Body Control of Series-Parallel Hybrid Robots

The Video illustrates the results of the paper Dennis Mronga, Shivesh Kumar, Frank Kirchner: "Whole-Body Control of Series-Parallel Hybrid Robots", Accepted for Publication: IEEE International Conference on Robotics and Automation (ICRA), 23.5.-27.5.2022, Philadelphia, 2022.

RH5: Motion Capture State Feedback for Real-Time Control of a Humanoid Robot

The Video illustrates the results of the paper Mihaela Popescu, Dennis Mronga, Ivan Bergonzani, Shivesh Kumar, Frank Kirchner: "Experimental Investigations into Using Motion Capture State Feedback for Real-Time Control of a Humanoid Robot", Accepted for Publication: MDPI Sensors Journal, Special Issue "Advanced Sensors Technologies Applied in Mobile Robot", 2022.

RH5 Manus: Background of robot dance generation based on music analysis driven trajectory optimization

RH5 Manus: Robot Dance Generation based on Music Analysis Driven Trajectory Optimization

Musical dancing is an ubiquitous phenomenon in human society. Providing robots the ability to dance has the potential to make the human robot co-existence more acceptable. Hence, dancing robots have generated a considerable research interest in the recent years. In this paper, we present a novel formalization of robot dancing as planning and control of optimally timed actions based on beat timings and additional features extracted from the music. We showcase the use of this formulation in three different variations: with input of a human expert choreography, imitation of a predefined choreography, and automated generation of a novel choreography. Our method has been validated on four different musical pieces, both in simulation and on a real robot, using the upper-body humanoid robot RH5 Manus.

RH5 Manus: Introduction of a Powerful Humanoid Upper Body Design for Dynamic Movements

Recent studies suggest that a stiff structure along with an optimal mass distribution are key features to perform dynamic movements, and parallel designs provide these characteristics to a robot. This work presents the new upper-body design of the humanoid robot RH5 named RH5 Manus, with series-parallel hybrid design. The new design choices allow us to perform dynamic motions including tasks that involve a payload of 4 kg in each hand,

and fast boxing motions. The parallel kinematics combined with an overall serial chain of the robot provides us with high force production along with a larger range of motion and low peripheral inertia. The robot is equipped with force-torque sensors, stereo camera, laser scanners, high-resolution encoders etc that provide interaction with operators and environment. We generate several diverse dynamic motions using trajectory optimization, and successfully execute them on the robot with accurate trajectory and velocity tracking, while respecting joint rotation, velocity, and torque limits.

RH5 Manus: Humanoid assistance robot for future space missions

The humanoid robot "RH5 Manus" was developed as part of the "TransFIT" project as an assistance robot that can be used in the direct human environment, for example on a future moon station. The aim was to equip the robot with the necessary capabilities to perform complex assembly work autonomously, as well as in cooperation with astronauts and teleoperated. Another focus of the project was on the transfer of the developed technologies to industrial manufacturing and production. The video shows the mechanical assembly and the commissioning of the robot.

RH5: Design, Analysis and Control of the Series-Parallel Hybrid RH5 Humanoid Robot

This paper presents a novel series-parallel hybrid humanoid called RH5 which is 2 m tall and weighs only 62.5 kg capable of performing heavy-duty dynamic tasks with 5 kg payloads in each hand. The analysis and control of this humanoid is performed with whole-body trajectory optimization technique based on differential dynamic programming (DDP). Additionally, we present an improved contact stability soft-constrained DDP algorithm which is able to generate physically consistent walking trajectories for the humanoid that can be tracked via a simple PD position control in a physics simulator. Finally, we showcase preliminary experimental results on the RH5 humanoid robot.

Torque-limited simple pendulum: A toolkit for getting started with underactuated robotics

This project describes the hardware (Computer-aided design (CAD) models, Bill Of Materials (BOM), etc.) required to build a physical pendulum system and provides the software (Unified Robot Description Format (URDF) models, simulation and controller) to control it. It provides a setup for studying established and novel control methods on a simple torque-limited pendulum, and targets students and beginners of robotic control. In this video we will cover mechanical and electrical setup of the test bed, introduce offline trajectory optimization methods and showcase model-based as well as data-driven controllers. The entire hardware and software description is open-source available.



Investigations into Exploiting the Full Capabilities of a Series-Parallel Hybrid Humanoid using Whole Body Trajectory Optimization
Melya Boukheddimi, Rohit Kumar, Shivesh Kumar, Carpentier Justin, Frank Kirchner
In 2023 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), (IROS-2023), 01.10.-05.10.2023, Detroit, Michigan, IEEE, Oct/2023.
End-to-End Reinforcement Learning for Torque Based Variable Height Hopping
Raghav Soni, Daniel Harnack, Hannah Isermann, Sotaro Fushimi, Shivesh Kumar, Frank Kirchner
In 2023 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), (IROS-2023), 01.10.-05.10.2023, Detroit, MI, IEEE Robotics and Automation Society, pages 7531-7538, Oct/2023. ISBN: 978-1-6654-9190-7.
AcroMonk: A Minimalist Underactuated Brachiating Robot
Mahdi Javadi, Daniel Harnack, Paula Stocco, Shivesh Kumar, Shubham Vyas, Daniel Pizzutilo, Frank Kirchner
In IEEE Robotics and Automation Letters, IEEE, volume 8, pages 1-8, Jun/2023.
A Recursive Lie-Group Formulation for the Second-Order Time Derivatives of the Inverse Dynamics of Parallel Kinematic Manipulators
Andreas Müller, Shivesh Kumar, Thomas Kordik
In IEEE Robotics and Automation Letters, IEEE, volume 8, pages 1-8, Apr/2023.
Deriving Rewards for Reinforcement Learning from Symbolic Behaviour Descriptions of Bipedal Walking
Daniel Harnack, Christoph Lüth, Lukas Groß, Shivesh Kumar, Frank Kirchner
In 62nd IEEE Conference on Decision and Control (CDC), (CDC-2023), 13.12.-15.12.2023, Marina Bay Sands, TBA., 2023.
Fast Dynamic Walking with RH5 Humanoid Robot
Ivan Bergonzani, Mihaela Popescu, Shivesh Kumar, Frank Kirchner
In 2023 IEEE-RAS 22nd International Conference on Humanoid Robots (Humanoids), (Humanoids), 12.12.-14.12.2023, Austin, TX, IEEE, 2023. ISBN: 979-8-3503-0327-8.


Analytic Estimation of Region of Attraction of an LQR Controller for Torque Limited Simple Pendulum
Lukas Groß, Lasse Jenning Shala, Shivesh Kumar, Frank Kirchner, Christoph Lüth
In 61st IEEE Conference on Decision and Control, (CDC-2022), 6.12.-9.12.2022, Cancun, Institute of Electrical and Electronics Engineers (IEEE), Dec/2022.
Experimental Investigations into Using Motion Capture State Feedback for Real-Time Control of a Humanoid Robot
Mihaela Popescu, Dennis Mronga, Ivan Bergonzani, Shivesh Kumar, Frank Kirchner
In Sensors - Open Access Journal, Multidisciplinary Digital Publishing Institute (MDPI), volume 22, number 24, pages 1-12, Dec/2022.
Modular and Hybrid Numerical-Analytical Approach - A Case Study on Improving Computational Efficiency for Series-Parallel Hybrid Robots
Rohit Kumar, Shivesh Kumar, Andreas Mueller, Frank Kirchner
In 2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), (IROS-2022), 23.10.-27.10.2022, Kyoto, IEEE, Nov/2022.
Torque-limited simple pendulum: A toolkit for getting familiar with control algorithms in underactuated robotics
Felix Wiebe, Jonathan Babel, Shivesh Kumar, Shubham Vyas, Daniel Harnack, Melya Boukheddimi, Mihaela Popescu, Frank Kirchner
In Journal of Open Source Software, The Open Journal, volume 7, number 74, pages 1-7, Jun/2022.
Kinematic Analysis of a Novel Humanoid Wrist Parallel Mechanism
Christoph Stoeffler, Adriano del Rio Fernandez, Heiner Peters, Moritz Schilling, Shivesh Kumar
Editors: Bruno Siciliano, Oussama Khatib
In ARK 2022: Advances in Robot Kinematics 2022, (ARK-2022), 27.6.-30.6.2022, Bilbao, Springer, series Springer Proceedings in Advanced Robotics, volume 24, pages 348-355, Jun/2022. ISBN: 978-3-031-08140-8.
A Survey on Design and Control of Lower Extremity Exoskeletons for Bipedal Walking
Ibrahim Tijjani, Shivesh Kumar, Melya Boukheddimi
In Applied Sciences, MDPI, volume 12, number 5, pages 1-31, Feb/2022.
Post-Capture Detumble Trajectory Stabilization for Robotic Active Debris Removal
Shubham Vyas, Lasse Maywald, Shivesh Kumar, Marko Jankovic, Andreas Mueller, Frank Kirchner
In Advances in Space Research, Elsevier Ltd., volume 1, pages 1-18, 2022.


Closed-form time derivatives of the equations of motion of rigid body systems
Andreas Müller, Shivesh Kumar
In Multibody System Dynamics, Springer, volume o.A., pages o.A., Jul/2021.
Design, Analysis and Control of the Series-Parallel Hybrid RH5 Humanoid Robot
Julian Eßer, Shivesh Kumar, Heiner Peters, Vinzenz Bargsten, José de Gea Fernández, Carlos Mastalli, Olivier Stasse, Frank Kirchner
In 2020 IEEE-RAS 20th International Conference on Humanoid Robots (Humanoids), (Humanoids-2020), 19.7.-21.7.2021, Munich/Virtual, IEEE, pages 400-407, Jul/2021.
Nth Order Analytical Time Derivatives of Inverse Dynamics in Recursive and Closed Forms
Shivesh Kumar, Andreas Mueller
In 2021 IEEE International Conference on Robotics and Automation (ICRA), (ICRA-2021), 30.5.-05.6.2021, Xi'an, IEEE, Jun/2021. IEEE.

Back to the list of projects
last updated 04.01.2024