DexSuite: An All-in-One Framework for Dexterous Manipulation Across Arms, Hands, and Tasks
Paper • 2610.37799 • Published
200 episodes · 20 fps · 2 cameras · 224×224 av1
This dataset was created using LeRobot.
Dataset Description
Scripted-expert DexSuite data for close_window_spray with the floating wrist and the Sharpa hand: 200 successful episodes (138,779 frames) from DexSuite's scripted task controller with grasp-recovery perturbations (DART) and randomized object and start-pose initialization, including per-finger contact forces. This task is not part of the DexSuite benchmark. A double-hung window stands on the table with its lower sash raised; the hand pushes the sash shut by its lift rail, picks up a trigger spray bottle, pulls the trigger with the index finger and sprays the closed pane. The task succeeds once the sash is closed (within 2 cm) and the spray has hit the glass on 20 control steps with the trigger pulled through at least 70% of its travel and the nozzle 14 to 36 cm from the glass. All episodes are successful, recorded at 20 Hz with a static front camera and a wrist camera (224x224 RGB video), proprioception and ground-truth object state, under DexSuite's shared observation/action interface (actions are OSC pose offsets from the episode-start end-effector pose plus gripper/hand joint targets). As in every DexSuite set, frame t stores the observation before action t; the success flag on the last frame refers to the transition it triggers, so the terminal state itself is not stored.
The controller closes the sash with one of two fingertip presses on the rail (paired fingertips on its left in 102 episodes, a relaxed press at its center in 98), wraps the bottle into the palm, carries it toward the pane, frees the index finger, sets it on the trigger and sprays from a sampled distance. It keeps spraying after the first success, so success and terminated are true on every frame from the first successful transition on (the last 40 frames of an episode at the median). DART: during recording the environment executed the controller's command plus a smooth Ornstein-Uhlenbeck perturbation of the wrist pose (1.5 cm and 4 degrees, 0.6 s time constant, scaled by 1.0 / 0.5 / 0.25 in the free / approach / in-hand phases and switched off while closing the sash, sliding the palm onto the bottle, carrying it and working the trigger), and the recorded action is the controller's clean command, its correction from the perturbed state (Laskey et al., 2017). The command the environment actually executed at step t is stored in obs.state.other.action of frame t+1, so replaying action open loop does not reproduce the recorded episodes. Each episode randomizes the bottle position (±10 cm in x and y) and yaw (±15°), the start wrist pose (±5 cm and ±12° per axis) and the relaxed finger pose; 200 of 499 attempts were kept, the other 299 failed the controller's own checks and were discarded.
The recording uses 8 physics substeps per control step (physics_substeps: 8 in meta/env_config.json; DexSuite's default is 2), so build evaluation environments from meta/env_config.json. The front camera is the window task's own view (window_front). Close Window and Spray is not in the PyPI package; clone the DexSuite repository to build its environment.
The robot is an arm-less floating wrist (a virtual 6-DoF gantry) carrying the right Sharpa hand (22 DoF); the 29-D action is 7 osc_pose_abs_quat values (position offset and orientation relative to the episode-start wrist pose) followed by 22 hand joint position targets in radians. The hand is sharpa_fardexbot, the right Sharpa hand carrying the wrist-camera mount of the real handheld rig (mount plate, quick changer and D405 body, which collide with the scene and show in the front-camera frames); DexSuite ships it under that name. Besides the standard DexSuite keys, every frame carries per-finger contact forces (obs.state.other.finger_force_world, finger_force_norm in N, finger_force_display, finger_force_saturated, finger_contact; thumb, index, middle, ring, pinky) and the controller's phase (obs.state.other.controller_phase: 0 free, 1 approach, 2 in hand) and stage (obs.state.other.controller_stage, named in meta/controller.json). meta/env_config.json rebuilds the environment and meta/controller_episodes/episode_*.json keeps each episode's scene seed, style seed and controller parameters. Depth was rendered during collection but is not included.
- Homepage: https://huggingface.co/dexsuite
- Paper: https://arxiv.org/abs/2610.37799
- License: apache-2.0
Loading
from lerobot.datasets.lerobot_dataset import LeRobotDataset
dataset = LeRobotDataset("dexsuite/close_window_spray_floating_sharpa_scripted")
frame = dataset[0] # action, obs.state.*, obs.cameras.front.rgb and obs.cameras.wrist.rgb (3x224x224 float in [0, 1])
Training
DexSuite's robomimic entry point converts the set to HDF5 and trains on it:
python -m scripts.imitation_learning.robomimic.train \
--dataset close_window_spray_floating_sharpa_scripted \
--config bc \
--sensors state \
--architecture mlp
--sensors cameras --cameras front trains on the front camera and robot proprio instead, and --config act, act_img or diffusion select the other policy recipes (see scripts/imitation_learning/robomimic/README.md). --dataset dexsuite/close_window_spray_floating_sharpa_scripted works as well.
Rollouts build the environment from meta/env_config.json, 8 physics substeps included.
Dataset Structure
{
"codebase_version": "v3.0",
"robot_type": null,
"total_episodes": 200,
"total_frames": 138779,
"total_tasks": 1,
"chunks_size": 1000,
"data_files_size_in_mb": 100,
"video_files_size_in_mb": 200,
"fps": 20,
"splits": {
"train": "0:200"
},
"data_path": "data/chunk-{chunk_index:03d}/file-{file_index:03d}.parquet",
"video_path": "videos/{video_key}/chunk-{chunk_index:03d}/file-{file_index:03d}.mp4",
"features": {
"obs.state.gripper.qpos": {
"dtype": "float32",
"shape": [
22
]
},
"obs.state.gripper.qtorque": {
"dtype": "float32",
"shape": [
22
]
},
"obs.state.gripper.qvel": {
"dtype": "float32",
"shape": [
22
]
},
"obs.state.gripper.tcp_pos": {
"dtype": "float32",
"shape": [
3
]
},
"obs.state.gripper.tcp_quat": {
"dtype": "float32",
"shape": [
4
]
},
"reward": {
"dtype": "float32",
"shape": [
1
]
},
"truncated": {
"dtype": "bool",
"shape": [
1
]
},
"terminated": {
"dtype": "bool",
"shape": [
1
]
},
"success": {
"dtype": "bool",
"shape": [
1
]
},
"failure": {
"dtype": "bool",
"shape": [
1
]
},
"action": {
"dtype": "float32",
"shape": [
29
]
},
"obs.state.manipulator.eef_pos": {
"dtype": "float32",
"shape": [
1,
3
]
},
"obs.state.manipulator.eef_quat": {
"dtype": "float32",
"shape": [
1,
4
]
},
"obs.state.manipulator.qpos": {
"dtype": "float32",
"shape": [
6
]
},
"obs.state.manipulator.qtorque": {
"dtype": "float32",
"shape": [
6
]
},
"obs.state.manipulator.qvel": {
"dtype": "float32",
"shape": [
6
]
},
"obs.state.other.action": {
"dtype": "float32",
"shape": [
29
]
},
"obs.state.other.bottle_pos": {
"dtype": "float32",
"shape": [
3
]
},
"obs.state.other.bottle_quat": {
"dtype": "float32",
"shape": [
4
]
},
"obs.state.other.bottle_vel": {
"dtype": "float32",
"shape": [
3
]
},
"obs.state.other.circle_complete": {
"dtype": "float32",
"shape": [
1
]
},
"obs.state.other.circle_sector_coverage": {
"dtype": "float32",
"shape": [
1
]
},
"obs.state.other.circle_spray_steps": {
"dtype": "float32",
"shape": [
1
]
},
"obs.state.other.last_action": {
"dtype": "float32",
"shape": [
29
]
},
"obs.state.other.latch_q": {
"dtype": "float32",
"shape": [
1
]
},
"obs.state.other.nozzle_dir": {
"dtype": "float32",
"shape": [
3
]
},
"obs.state.other.nozzle_pos": {
"dtype": "float32",
"shape": [
3
]
},
"obs.state.other.paint_debug_spheres": {
"dtype": "float32",
"shape": [
1
]
},
"obs.state.other.pane_center": {
"dtype": "float32",
"shape": [
3
]
},
"obs.state.other.pane_coverage": {
"dtype": "float32",
"shape": [
1
]
},
"obs.state.other.pane_normal": {
"dtype": "float32",
"shape": [
3
]
},
"obs.state.other.pane_spray_complete": {
"dtype": "float32",
"shape": [
1
]
},
"obs.state.other.pane_spray_steps": {
"dtype": "float32",
"shape": [
1
]
},
"obs.state.other.pull_frac": {
"dtype": "float32",
"shape": [
1
]
},
"obs.state.other.sash_closed": {
"dtype": "float32",
"shape": [
1
]
},
"obs.state.other.sash_frac": {
"dtype": "float32",
"shape": [
1
]
},
"obs.state.other.sash_pos": {
"dtype": "float32",
"shape": [
3
]
},
"obs.state.other.sash_q": {
"dtype": "float32",
"shape": [
1
]
},
"obs.state.other.spray_active": {
"dtype": "float32",
"shape": [
1
]
},
"obs.state.other.trigger_q": {
"dtype": "float32",
"shape": [
1
]
},
"obs.cameras.front.rgb": {
"dtype": "video",
"shape": [
224,
224,
3
],
"info": {
"video.height": 224,
"video.width": 224,
"video.codec": "av1",
"video.pix_fmt": "yuv420p",
"video.is_depth_map": false,
"video.fps": 20,
"video.channels": 3,
"has_audio": false
}
},
"obs.cameras.wrist.rgb": {
"dtype": "video",
"shape": [
224,
224,
3
],
"info": {
"video.height": 224,
"video.width": 224,
"video.codec": "av1",
"video.pix_fmt": "yuv420p",
"video.is_depth_map": false,
"video.fps": 20,
"video.channels": 3,
"has_audio": false
}
},
"obs.state.other.finger_force_world": {
"dtype": "float32",
"shape": [
5,
3
]
},
"obs.state.other.finger_force_norm": {
"dtype": "float32",
"shape": [
5
]
},
"obs.state.other.finger_force_display": {
"dtype": "float32",
"shape": [
5
]
},
"obs.state.other.finger_force_saturated": {
"dtype": "bool",
"shape": [
5
]
},
"obs.state.other.finger_contact": {
"dtype": "bool",
"shape": [
5
]
},
"obs.state.other.controller_phase": {
"dtype": "int64",
"shape": [
1
]
},
"obs.state.other.controller_stage": {
"dtype": "int64",
"shape": [
1
]
},
"timestamp": {
"dtype": "float32",
"shape": [
1
],
"names": null
},
"frame_index": {
"dtype": "int64",
"shape": [
1
],
"names": null
},
"episode_index": {
"dtype": "int64",
"shape": [
1
],
"names": null
},
"index": {
"dtype": "int64",
"shape": [
1
],
"names": null
},
"task_index": {
"dtype": "int64",
"shape": [
1
],
"names": null
}
},
"task": "close_window_spray",
"options": null
}
Citation
BibTeX:
[More Information Needed]
- Downloads last month
- 61