For each grasp, there are a large number of possible contact points with different properties. The quality of grasps can be determined using metrics. Metrics are tools that can be used to measure a specific characteristic of a process. To mimic human grasping, these metrics could be used. However, to measure the quality of grasps and contact points, it is often unclear what metrics are needed.
Therefore, it is important to find out what metrics are required when robotic hands are used to mimic human grasping. The use of monocular RGB recordings would also allow a variety of data from video platforms to be used. More specifically, the contact areas and forces during the human grasping process should be analyzed and determined.
There are several approaches for contact point detection and force estimation based on either wearables or vision. Wearable-based methods use tactile multisensor gloves that can measure both inertia and flexion. The main drawback here is the relatively large setup required for the measurement. For this reason, it is more difficult to apply them to robotic hands later.
Vision-based methods use motion capture or images with or without depth information. While the setup is relatively small, monocular RGB-based methods tend to be inaccurate if there are occlusions over the hand in an image. Force estimation with image-based methods is also difficult.
The new approach of this project is to incorporate the deformability of objects and extract 3D poses and translations of hand and object from monocular RGB with relatively little influence of occlusions.
Thus, the focus is less on the hand and more on the interaction between hand and object. Meshes of hand and object are created, analyzed using a contact model, and metrics are calculated on this basis.
Datasets and meshes have been and will be tested. The preferred dataset is the HO3D dataset of Hampali et al.(2021) and the kypt_transformer of Hampali et al.(2022).
If monocular RGB provides too little information, RGBD or motion capture data can be used.
Key points detected in the image are transferred to 2D surface meshes such as YCB object meshes and the MANO model, and then transformed to 3D volume meshes. The volume meshes are used to estimate contact areas from intersections between meshes.
The pressure field model with hydroelastic contacts is used to estimate forces that can be used to evaluate metrics.