Behaviors for Mobile Manipulation

The main goal of the project BesMan is the development of generic dexterous manipulation strategies which are independent of a specific robot morphology. This will be done by integrating three key components: namely, trajectory planning, whole-body sensor-based reactive control, and dynamic control into a modular, robot-independent and easily-reconfigurable software framework which allows reusing components to describe a variety of complex manipulation behaviours. Furthermore, novel, situation-specific behavior shall be learned by means of a highly-automated machine learning platform, which incorporates an interface to a human operator, who via demonstration will show the robot how to deal with unforeseen situations. The development of dexterous manipulation procedures is mainly the responsibility of the DFKI RIC while the University of Bremen develops the machine learning platform.

Duration: 01.05.2012 till 31.07.2016
Donee: German Research Center for Artificial Intelligence GmbH & University of Bremen
Sponsor: Federal Ministry of Economics and Technology
German Aerospace Center e.V.
Grant number: This project is funded by the German Space Agency (DLR Agentur) with federal funds of the Federal Ministry of Economics and Technology in accordance with the parliamentary resolution of the German Parliament, grant no. 50 RA 1216 (DFKI) and 50 RA 1217 (University of Bremen).
Partner: Robotics Group, University of Bremen
Application Field: Space Robotics
Related Projects: LIMES
Learning Intelligent Motions for Kinematically Complex Robots for Exploration in Space (05.2012- 04.2016)
A Semi-Autonomous Free-Climbing Robot for the Exploration of Crater Walls and Bottoms (07.2007- 11.2010)
Intelligent Human-Robot Collaboration (03.2015- 06.2016)
Related Robots: AILA
Mobile Dual-Arm-Manipulation
Compliant Robot Arm
Exoskeleton Passive (CAPIO)
Upper body Human-Machine-Interface (HMI) for tele-operation
iMRK Dual-Arm Robot (Photo: Annemarie Popp, DFKI GmbH)
Related Software: Bagel
Biologically inspired Graph-Based Language
Behavior Optimization and Learning for Robots
Node Level Data Link Communication

Project details

As shown in neuroscience studies, human manipulative skills are not the result of high powerful actuation systems or fast sensorimotor loops. On the contrary, the key lies on the task organization, involving sequences of organized action plans at various levels of complexity.  We embraced this concept in the project BesMan and defined an ,Action Plan' as a sequence of subtasks which needed to be followed in order to accomplish a certain complex  task (for instance, building a specific infrastructure on the Moon). Thus, the subtasks of the ,Action Plan' can be mapped to behaviour templates available on the robotic system. The mapping between those subtasks and behaviour templates would be pre-defined, however, those templates are to some degree adaptive and will adapt to a certain situation when executed.

In order to fulfill such a generality, a software framework was developed to describe and control robot manipulation behaviors. To keep independence from particular robot hardware and an explicit statement of areas of application, an embedded domain specific language (eDSL) was used to describe the particular robot and a controller network that drives the robot. Thus, we made use of a) a component-based software framework, b) model-based algorithms for motion- and sensor processing representations, c) an abstract model of the control system, and d) a plan management software to describe a sequence of software component networks that generate the desired robot behavior.

In the area of robot dynamic control, we developed methods to obtain robot dynamic models from experimental identification by generating appropriate identification experiments and parameter estimation from the measurements. A software library was developed which builds first a parametrized dynamic model by using the known robot geometry. The next component of the library contains methods for experimental identification of those unknown  dynamic parameters. This includes a library to generate identification trajectories (taking into account joint limits, maximum velocities and accelerations, for instance) and a component for the estimation of the parameters from the experimental measurements. Once the model is available, the dynamic controller can be implemented.

In the area of manipulation planning, we created a sample-based motion planning library based on OMPL and in a similar fashion as MoveIt! as the main pipeline. Additionally, we included some functionality for dual-arm operations with Cartesian constraints, self-filtering and the possibility to plan in dynamic environments via replanning.

In the area of whole-body control, we used a constraint-based Whole Body Control scheme for executing reactive robot motions, incorporating the physical constraints of a system, integrating multiple, disparate sensor-based controllers and optimally utilizing the redundant degrees of freedom in complex systems, like e.g. AILA. In particular we integrated a constraint-based, multi-objective robot controller similar to iTAsC in our ROCK framework and provided the infrastructure to execute action plans based on motion constraints, as well as change the constraints online. All the active motion constraints and their parametrizations compose a subtask in our action plan that is executed by the robot.

In the area of the learning , we developed a learning platform which allows learning robotic manipulation behavior from human demonstrations. The whole procedure is highly automated and fast enough to serve as a  solution to quickly learn behavior for solving unforeseen challenges and tasks, which can always occur during deployment of a robotic system.  The learning platform is composed of several components, which deal with acquisition and preprocessing of human demonstrations, segmenting the demonstrated behavior into basic building blocks, learning movement primitives using imitation-learning, refining movement primitives by means of reinforcement learning, and generalizing movement primitives to related tasks.

The results of this project were developed as generic solutions which were independent of the morphology of the robot and of its application scenario. The transfer of the developed solutions between different robotic systems and between application scenarios (e.g. space and logistics) was exemplary demonstrated, as it can be seen in the different videos below.


BESMAN: Third Demonstration - MANTIS robot

Third official demonstration: Execution of an autonomous manipulation task in a space scenario and learning of manipulation behaviour from human demonstration.

BESMAN: Second Demonstration - KUKA iiwa robot

Second official demonstration: Learning reaching behaviour from human demonstration in a logistics scenario.

BesMan: First demonstration – robot system AILA

First official demonstration: Execution of an autonomous manipulation task in the ISS Mockup



Detection and recognition of human manipulation building blocks
Lisa Gutzeit
Mar/2023. Universität Bremen.


Learning Task Constraints for Whole-Body Control of Robotic Systems
Dennis Mronga
May/2022. Universität Bremen.


Distributed computation and control of robot motion dynamics on FPGAs
Vinzenz Bargsten, José de Gea Fernández
In SN Applied Sciences, Springer Nature, volume 2, number 7, pages 1239-n.n., Jun/2020.
BOLeRo: Behavior Optimization and Learning for Robots
Alexander Fabisch, Malte Langosz, Frank Kirchner
In International Journal of Advanced Robotic Systems, SAGE Publications, volume 17, number 3, pages n.n.-n.n., 2020.


Simple and Robust Automatic Detection and Recognition of Human Movement Patterns in Tasks of Different Complexity
Lisa Gutzeit, Marc Otto, Elsa Andrea Kirchner
Editors: Andreas Holzinger, Alan Pope, Hugo Plácido Silva
In Physiological Computing Systems, Springer, pages 39-57, Jul/2019.
pytransform3d: 3D Transformations for Python
Alexander Fabisch
In Journal of Open Source Software, The Open Journal, Journal of Open Source Software, volume 4, number 33, pages 1159, Jan/2019.


The BesMan Learning Platform for Automated Robot Skill Learning
Lisa Gutzeit, Alexander Fabisch, Marc Otto, Jan Hendrik Metzen, Jonas Hansen, Frank Kirchner, Elsa Andrea Kirchner
In Frontiers in Robotics and AI, o.A., volume 5, pages 43, May/2018.


Manipulation and Control
José de Gea Fernández, Elie Allouis, Karol Seweryn, Frank Kirchner, Yang Gao
Editors: Yang Gao
In Contemporary Planetary Robotics: An Approach Toward Autonomous Systems, Wiley-VCH Verlag GmbH & Co. KGaA, chapter 5, pages 255-319, Aug/2016. ISBN: 978-3-527-41325-6.
Minimum Regret Search for Single- and Multi-Task Optimization
Jan Hendrik Metzen
In International Conference on Machine Learning, (ICML), 19.6.-24.6.2016, New York, o.A., Jun/2016.
Experimental Robot Inverse Dynamics Identification Using Classical and Machine Learning Techniques
Vinzenz Bargsten, José de Gea Fernández, Yohannes Kassahun
In International Symposium on Robotics, (ISR), 21.6.-22.6.2016, München, o.A., 2016.
Automatic Detection and Recognition of Human Movement Patterns in Manipulation Tasks
Lisa Gutzeit, Elsa Andrea Kirchner
In Proceedings of the 3rd International Conference on Physiological Computing Systems, (PHYCS-16), 27.7.-28.7.2016, Lissabon, SciTePress, pages 54-63, 2016.
Zusammen mit Robotern am Arbeitsplatz - Neue Bedienkonzepte eröffnen den Weg für neue Robotikanwendungen
Malte Wirkus, Vinzenz Bargsten
In Industrie 4.0 Management, Gito mbH Verlag für Industrielle Informationstechnik und Organisation, volume 2, pages 29-32, 2016.


Active Contextual Entropy Search
Jan Hendrik Metzen
In 8th Workshop on Optimization for Machine Learning (OPT 2015), (OPT-2015), 11.12.2015, Montreal, o.A., Dec/2015.
Single and Dual Arm Manipulator Motion Planning Library
Behnam Asadi
In Workshop on Task Planning for Intelligent Robots in Service and Manufacturing, (IROS), 2015, o.A., Oct/2015.
Intuitive Interaction with Robots - Technical Approaches and Challenges
Elsa Andrea Kirchner, José de Gea Fernández, Peter Kampmann, Martin Schröer, Jan Hendrik Metzen, Frank Kirchner
In Formal Modeling and Verification of Cyber Physical Systems, Springer Heidelberg, pages 224-248, Sep/2015. ISBN: 978-3-658-09993-0.
Accounting for Task-Difficulty in Active Multi-Task Robot Control Learning
Alexander Fabisch, Jan Hendrik Metzen, Mario Michael Krell, Frank Kirchner
Editors: Hermann Engesser
In KI - Künstliche Intelligenz, German Journal on Artificial Intelligence - Organ des Fachbereiches "Künstliche Intelligenz" der Gesellschaft für Informatik e.V., Springer Verlag GmbH Heidelberg, the final publication is available at, volume 29, pages 369-377, 2015.
COMPI: Development of a 6-DOF Compliant Robot Arm for Human-Robot Cooperation
Vinzenz Bargsten, José de Gea Fernández
In Proceedings of the 8th International Workshop on Human-Friendly Robotics, (HFR-2015), 21.10.-23.10.2015, Munich, o.A., 2015. Technische Universität München (TUM).
Towards Describing and Deploying Whole-Body Generic Manipulation Behaviours
José de Gea Fernández, Dennis Mronga, Malte Wirkus, Vinzenz Bargsten, Behnam Asadi, Frank Kirchner
In 2015 Space Robotics Symposium, 29.10.-30.10.2015, Glasgow, IET, University of Strathclyde, 2015.
Bayesian Optimization for Contextual Policy Search
Jan Hendrik Metzen, Alexander Fabisch, Jonas Hansen
In Proceedings of the Second Machine Learning in Planning and Control of Robot Motion Workshop, (IROS MLPC-2015), 02.10.2015, Hamburg, IROS, 2015.


Towards Robot-independent Manipulation Behavior Description
Malte Wirkus
In Proceedings of the 5th International Workshop on Domain-Specific Languages and models for ROBotic systems, (DSLRob-2014), 20.10.2014, Bergamo, o.A., Oct/2014.
Velocity-Based Multiple Change-point Inference for Unsupervised Segmentation of Human Movement Behavior
Lisa Gutzeit, Martin Schröer, Jan Hendrik Metzen, Elsa Andrea Kirchner
In Proceedings of the 22nd International Conference on Pattern Recognition, (ICPR-2014), 24.8.-28.8.2014, Stockholm, IEEE, pages 4564-4569, Aug/2014.
Towards Learning of Generic Skills for Robotic Manipulation
Jan Hendrik Metzen, Alexander Fabisch, Lisa Gutzeit, José de Gea Fernández, Elsa Andrea Kirchner
In KI - Künstliche Intelligenz, German Journal on Artificial Intelligence - Organ des Fachbereiches "Künstliche Intelligenz" der Gesellschaft für Informatik e.V., Springer, volume 28, number 1, pages 15-20, Mar/2014.
Active Contextual Policy Search
Alexander Fabisch, Jan Hendrik Metzen
In Journal of Machine Learning Research, Microtome Publishing, volume 15, pages 3371-3399, 2014.


Learning Graph-based Representations for Continuous Reinforcement Learning Domains
Jan Hendrik Metzen
In Proceedings of the European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases, (ECML PKDD-2013), 23.9.-27.9.2013, Prag, Springer Verlag GmbH, pages 81-96, Sep/2013.
Learning Skill Templates for Parameterized Tasks
Jan Hendrik Metzen, Alexander Fabisch
In Proceedings of the 11th European Workshop on Reinforcement Learning, (EWRL-13), 04.8.-09.8.2013, Dagstuhl, o.A., Aug/2013.
Online Skill Discovery using Graph-based Clustering
Jan Hendrik Metzen
In Journal of Machine Learning Research, o.A., volume W&CP 24, pages 77-88, 2013.
Incremental Learning of Skill Collections based on Intrinsic Motivation
Jan Hendrik Metzen, Frank Kirchner
In Frontiers in Neurorobotics, o.A., volume 7, number 11, pages 1-12, 2013.

Back to the list of projects
last updated 04.01.2024
to top