MetaIsaacGrasp: IsaacLab for Supervised / Reinforcement learning

A data generation engine and test-bench for grasp learning including:

• Data generation AIR-v0-Data
• Policy evaluation AIR-v0-Grasp
• Teleoperation AIR-v0-Tele
• Reinforcement learning (In progress)AIR-v0-Cont

Click to watch the demo video (Inference with remote agent functionality by vMF-Contact)

(All the successfully grasped objects will be put under the table.)

Teleoperation

Our teleoperation may also supported by vMF-Contact to reach objects:

New features compared to old version:

1. Now support collision check and visualization of non-collision grasps (adapted from MGN)

Non-collision grasp generation pipeline

2. Both one-grasp-per-image and continuous learning supported
3. Camera on hand, top-down and random / equally distributed camera view on the hemisphere.
4. NVIDIA Warp for state machine (state machine supported by NVIDIA Warp).
5. Teleoperation environment (AIR-v0-Tele)
6. Remote grasp agent to work around the environment conflict (see vMF-Contact))

Current state

Captured images: RGB, Segmentation, Depth, Normals

Getting started

Make sure you already installed the Isaac-Sim in proper manner.

Install Isaac Lab following the installation tutourial. Please make sure that is under your home directory: ~/IsaacLab and following environment variables are added into .bashrc:

# Isaac Sim root directory
export ISAACSIM_PATH="${HOME}/.local/share/ov/pkg/isaac-sim-4.2.0"
# Isaac Sim python executable
export ISAACSIM_PYTHON_EXE="${ISAACSIM_PATH}/python.sh"
# Isaac Lab path
export LAB_PATH="${HOME}/IssacLab$LAB_PATH"
# Add Isaac Lab's sh path for convenience when runing: isaaclab -p *.py
alias isaaclab=/home/{user_name}/IsaacLab/isaaclab.sh

MetaGraspNet objects available

Objects are now adapted from models. Unzip under the same directory and run isaaclab -p urdf_converter.py to convert all URDF files into USD files. We don't use original USD files since all the collision meshes are in convex hall, which are unrealistic.

Create symbolic link to your isaac sim by:

ln -s path_to_isaac_sim _isaac_sim

This is adapted from tutourial for binary installation

Now Isaaclab can also be used with Isaac Sim installed by pip. Please check the above documentation for details.

VSCode development

Now the vscode debugging is supported by pressing Ctrl+Shift+P, selecting Tasks: Run Task and run setup_python_env

You can change to headless mode as you wish. The num_envs decide how many scenes will be set up on the same stage.

Potential Issues

If you cause core dump due to camera setting please run following command:

sudo prime-select nvidia

in case your

sudo prime-select query

returns on-demand