DefGraspSim: A pipeline to evaluate robotic grasping of 3D deformable objects

Over the past few decades, roboticists and computer scientists have developed robots that can grasp and manipulate various objects in their surroundings. Most of these robots are primarily trained to grasp rigid objects or objects with specific shapes.

Most objects in the real world, however, including clothes, plastic bottles, or food items, are deformable, which essentially mean that they can easily change shape while someone is manipulating them. Training robots to grasp both rigid and deformable 3D objects is a crucial step towards the employment of robot for a variety of real-world applications, including food processing, robotic surgery or household assistance.

Although techniques that allow robots to grasp rigid objects have become increasingly advanced over the past few years, strategies for grasping these objects do not always transfer well to deformable objects. For instance, while a soft toy could be grasped haphazardly, a rigid object that does not conform to a user’s hand might require a steadier and more precise grasp. Similarly, while a rigid food container could be grasped strongly and decisively, if it were flexible a robot would need to be more careful to avoid crushing both the container and the food inside it.

Researchers at NVIDIA have recently developed DefGraspSim, a platform that can be used to evaluate strategies for grasping deformable objects using robotic hands or manipulators. This valuable platform, presented in a paper pre-published on arXiv, could help to improve the performance of robots designed to manipulate objects in real world settings.

“Creating grasp strategies for deformable objects has historically been difficult due to the complexities in modeling their physical responses,” Isabella Huang, Yashraj Narang, Clemens Eppner, Balakumar Sundaralingam, Miles Macklin, Tucker Hermans and Dieter Fox, the researchers who carried out the study, told TechXplore via email. “Only in recent years have fast and accurate robotic simulators been developed to address this issue. We leveraged one such simulator, Isaac Gym, to create DefGraspSim, a pipeline that allows researchers to automatically evaluate grasps of their choosing on their own custom objects.”

The platform developed by Huang and her colleagues allows researchers to carry out customized grasping experiments evaluating the performance of robots on manipulation tasks associated with specific domains, such as household, healthcare or agricultural settings. DefGraspSim could be a highly valuable platform for robotics research, seeing as many of the tasks it evaluates robots on would be challenging or unsafe for a robot to be tested on in the real world (e.g., those associated with surgical procedures or manufacturing).

In addition to using the platform to evaluate grasping strategies or grasp planners, researchers can use it to generate datasets containing grasp strategies. These datasets could then be used to train deep learning algorithms or other computational methods for identifying effective grasping strategies.

“Our work was motivated by the numerous exciting challenges that come with understanding how to grasp deformable objects,” Huang and her colleagues said. “Compared to the domain of rigid objects, which has received over 30 years of attention in the past, the study of deformable objects is heavily underexplored.”

In their paper, Huang and her colleagues focus on two main research questions. Firstly, the researchers wanted to determine how researchers can measure and assess the performance of deformable object grasps. Secondly, they wanted to devise a tool that would allow roboticists and computer scientists to measure and analyze these metrics in a reliable way.

“In the literature for rigid object grasping, the set of general performance metrics (i.e., measures that quantify how good a grasp is), is mostly unified across works,” the researchers explained. “Under a certain grasp, there are two major metrics of concern: whether an object can be picked up (grasp success), and whether that object can resist perturbations afterwards (grasp stability). While grasp success and stability apply to deformable objects as well, we also propose additional metrics that uniquely capture the responses of deformable objects.”

The additional metrics for capturing the responses of deformable objects proposed by Huang and her colleagues include deformation, stress, strain energy and deformation controllability. Deformation essentially quantifies how an object’s shape changes when it is grasped. Stress is a measure summarizing the stresses applied on an object’s body by the robotic gripper as it grasps it, which ultimately induces deformation. Notably, a stress measure that exceeds a material’s limits could lead to the object being permanent deformed, damaged or fractured.

Strain energy, on the other hand, is a measure summarizing the elastic potential energy stored in an object when it is grasped. Finally, deformation controllability refers to how much additional deformation an object can undergo, based on gravity, after the gripper is re-oriented.

Interestingly, the four additional metrics considered by the researchers can compete with one another. This means that, for instance, a grasp with high stability could lead to low deformation for one object and high deformation for another, depending on the objects’ composition, shape and structure.

“These metrics are comprehensive, so that practitioners can choose to evaluate what matters most for them,” Huang and her colleagues said. “For example, when grasping a block of tofu, one may want to select a low-stress grasp to make sure that it does not break. However, if one wants to use a ketchup bottle, one may choose a high deformation grasp so that the ketchup can more quickly be squeezed out. On the other hand, a low-deformation grasp may be best on a box of crackers so that the contents would not be crushed.”

Shortly after they started conducting their research, Huang and her colleagues realized that different researchers and roboticists might prioritize different metrics, depending on the robot they are testing or the evaluations they are planning to conduct. They thus tried to devise a platform that would allow users to analyze the metrics according to their unique domains of interest.

So far, most of the metrics outlined by this team of researchers have been extremely difficult to access in real-world settings, particularly field quantities such as stress and deformation. The team thus used a finite element method (FEM)-based simulator (Isaac Gym) as the best proxy for the ground truth, in instances where one has full access to an object’s physical state and all the metrics can be easily measured.

“Unlike classic rigid body model-based simulators (e.g., GraspIt! and OpenGRASP), Isaac Gym explicitly models deformation and stress dynamics and large kinematic and kinetic perturbations,” Huang and her colleagues said. “To enable other researchers to evaluate performance metrics on their own objects, we use Isaac Gym to build DefGraspSim, the first deformable grasping tool and database.”

DefGraspSim is a comprehensive and automatic grasp evaluation pipeline that researchers can use to automatically evaluate any of the performance metrics for arbitrary robotic grasps, focusing on their own customized deformable objects. Huang and her colleagues hope that their platform will soon become the preferred experimental environment for training computational models on the robotic grasping and manipulation of deformable objects in simulations.

Users simply need to input a 3D mesh (a specific kind of geometric model) of an object of their choice, along with strategies for grasping this object. Based on what they are trying to achieve, they can then pick between four different grasp evaluations, each measuring a subset of the total performance metrics. In addition, users can choose to customize the metrics they wish to focus their evaluation on or design and measure additional pre-pickup quantities, which the team refers to as ‘grasp features.”

“DefGraspSim is the first publicly released pipeline for grasping deformable objects,” Huang and her colleagues said. “In addition to creating this tool, we have also published a live dataset of full metric evaluations for grasps on 34 objects, most of which are modeled from scans of real deformable objects. We also include detailed visualizations of grasping results on several object primitives so that readers have a clear idea of what quantities can be extracted from the pipeline and can build physical intuition about how deformable objects respond under grasps.”

Huang and her colleagues carried out a series of experiments in real world settings aimed at validating the accuracy of Isaac Gym and the DefGraspSim pipeline. Their findings were highly promising, as they suggest that the results achieved on their simulation platform are similar to those achieved in real life.

The researchers’ paper won the Best Paper Award at the Workshop on Deformable Object Simulation in Robotics at Robotics Science and Systems (RSS) 2021. In addition, other research groups have already started using the DefGraspSim pipeline to evaluate their techniques for deformable object grasping.

“We believe that DefGraspSim is a very practical, all-purpose tool that can be customized to collect any metric or additional data that researchers would want,” Huang and her colleagues said. “There are many impactful ways in which we think our pipeline will provide a strong foundation for future work.”

In the future, DefGraspSim could enable more rigorous and reliable evaluations of tools for robotic object grasping, as well as comparisons between their performance in simulations and reality. The platform could also be extended to include vision-based measurements, such as those present in RGB-D images, or to build grasp planners that can generate optimal grasp strategies based on object properties and selected metrics.

“Currently, we are leveraging DefGraspSim to generate grasping experimental data of scale for the training of deep networks,” the researchers wrote. “We are using DefGraspSim to measure and generate high-dimensional features that will be used in a deep-learning framework for fast metric prediction on unseen objects.”

© 2021 Science X Network