This is the codebase for the simulation (AP2AP) part of our paper Dex4D. This repository contains code for the simulation (AP2AP) environment, teacher and student policy training, policy inference, and trained model checkpoints.

[Paper] [Project Page] [Checkpoints] [Code (Vision)] [Code (Hardware)]

The code is tested on Python 3.8.20 with cuda toolkit 11.8.

Please follow the steps below to perform the installation:

conda create -n dex4d-sim python==3.8.20
conda activate dex4d-sim

First install PyTorch:

pip install torch==2.0.1 torchvision==0.15.2 torchaudio==2.0.2 --index-url https://download.pytorch.org/whl/cu118

Then install pytorch3d:

pip install "git+https://github.com/facebookresearch/pytorch3d.git@stable" 

Then download Isaac Gym from the official source. After downloading and unzipping, install Isaac Gym in editable mode:

cd <PATH_TO_ISAACGYM_INSTALL_DIR>/python
pip install -e .

Once Isaac Gym is installed and samples work within your current python environment, install this repo from source code:

cd dex4d_policy/
pip install -e .

git clone [email protected]:erikwijmans/Pointnet2_PyTorch.git && cd Pointnet2_PyTorch/pointnet2_ops_lib/
python setup.py install

Please follow UniDexGrasp to set up the object dataset in dex4d_policy/assets folder.

The dex4d_policy/assets folder should look like:

dex4d_policy/assets
├── datasetv4.1
├── meshdatav3_pc_feat
├── meshdatav3_scaled
├── mjcf
├── textures
└── urdf

All the training commands are executed from the dex4d_policy/dex4d folder.

First fill in the <YOUR_LOG_DIR> in script/run_train_ppo_stage_1_2.sh.

# override config for stage 1 & 2 training
cp cfg/xarm6_leap_hand_ap2ap_stage_1_2.yaml cfg/xarm6_leap_hand_ap2ap.yaml
cp cfg/ppo/config_stage_1_2.yaml cfg/ppo/config.yaml
# launch training
bash script/run_train_ppo_stage_1_2.sh

First fill in the <YOUR_LOG_DIR> and <CKPT_NAME_FROM_STAGE_1_2> in script/run_train_ppo_stage_3.sh.

# override config for stage 3 training
cp cfg/xarm6_leap_hand_ap2ap_stage_3.yaml cfg/xarm6_leap_hand_ap2ap.yaml
cp cfg/ppo/config_stage_3.yaml cfg/ppo/config.yaml
# launch training
bash script/run_train_ppo_stage_3.sh

First fill in the <YOUR_LOG_DIR> and <CKPT_NAME_FROM_STAGE_3> in script/run_train_dagger.sh.

bash script/run_train_dagger.sh

For more provided args, please check these scripts and utils/config.py.

All the playing commands are executed from the dex4d_policy/dex4d folder, and pretrained models are provided in dex4d_policy/dex4d/example_models.

# First fill in <CKPT_NAME> in script/play.sh and script/play_dagger.sh, then run:
bash script/play.sh # Teacher PPO policy
bash script/play_dagger.sh # Student DAgger policy

This project is built upon the following open-source projects:

• PKU-EPIC/UniDexGrasp
• NVIDIA-Omniverse/IsaacGymEnvs
• PKU-MARL/DexterousHands

We thank the authors for their open-source contributions.

If you find this work helpful, please consider citing:

@article{kuang2026dex4d,
  title={Dex4D: Task-Agnostic Point Track Policy for Sim-to-Real Dexterous Manipulation},
  author={Kuang, Yuxuan and Park, Sungjae and Fragkiadaki, Katerina and Tulsiani, Shubham},
  journal={arXiv preprint arXiv:2602.15828},
  year={2026}
}