Differentiable Inverse Graphics for Zero-Shot Scene Reconstruction and Robot Grasping

Zusammenfassung (Abstract) :

Operating effectively in novel real-world environments requires robotic systems to estimate and interact with previously unseen objects. Current state-of-the-art models address this challenge by using large amounts of training data and test-time samples to build black-box scene representations. In this work, we introduce a differentiable neuro-graphics model that combines neural foundation models with physics-based differentiable rendering to perform zero-shot scene reconstruction and robot grasping without relying on any additional 3D data or test-time samples. Our model solves a series of constrained optimization problems to estimate physically consistent scene parameters, such as meshes, lighting conditions, material properties, and 6D poses of previously unseen objects from a single RGBD image and bounding boxes. We evaluated our approach on standard model-free fewshot benchmarks and demonstrated that it outperforms existing algorithms for model-free few-shot pose estimation. Furthermore, we validated the accuracy of our scene reconstructions by applying our algorithm to a zero-shot grasping task. By enabling zero-shot, physically-consistent scene reconstruction and grasping without reliance on extensive datasets or test-time sampling, our approach offers a pathway toward more data efficient, interpretable and generalizable robot autonomy in novel environments.

Stichworte :

Pixel;Radio access networks;Regional area networks;Location awareness;Unsolicited e-mail;Videos;Communication systems;Digital images;Mobile communication;Telecommunications;Cognitive modeling;RGB-D perception;perception for grasping and manipulation

Files:

20260604_differentiable_inverse_graphics_for_zero_shot_scene_reconstruction_and_robot_grasping.pdf

Links:

https://ieeexplore.ieee.org/document/11495081