DexSuite: An All-in-One Framework for Dexterous Manipulation Across Arms, Hands, and Tasks
Paper • 2610.37799 • Published
Datasets:
200 episodes · 20 fps · 2 cameras · 224×224 av1
This dataset was created using LeRobot.
Dataset Description
Scripted-expert DexSuite data for put_bread_toaster with the floating wrist and the Sharpa hand: 200 successful episodes (65,523 frames) from DexSuite's scripted task controller with grasp-recovery perturbations (DART) and randomized object and start-pose initialization, including per-finger contact forces. These are the Sharpa Put Bread in Toaster demonstrations of the DexSuite benchmark (Track 2). The hand pinches a slice of toast lying flat on the table, lifts it, stands it in a slot of the toaster and pushes the lever down; the task succeeds once the toast stands upright in a slot at least 3 cm below the rim, the lever is pressed through at least 80% of its travel and the toast is at rest (below 5 cm/s). All episodes are successful, recorded at 20 Hz with a static front camera and a wrist camera (224x224 RGB video), proprioception and ground-truth object state, under DexSuite's shared observation/action interface (actions are OSC pose offsets from the episode-start end-effector pose plus gripper/hand joint targets). As in every DexSuite set, frame t stores the observation before action t; the success flag on the last frame refers to the transition it triggers, so the terminal state itself is not stored.
The controller reaches the toast, pinches it, lifts it, carries it over a slot, lowers it in, releases it, moves over the lever and presses it down. Each episode ends at its first successful transition, so success and terminated are set on the last frame only. DART: during recording the environment executed the controller's command plus a smooth Ornstein-Uhlenbeck perturbation of the wrist pose (1.5 cm and 4 degrees, 0.6 s time constant, scaled by 1.0 / 0.5 / 0.1 in the free / approach / in-hand phases and switched off while pinching, lowering into the slot, releasing and pressing the lever), and the recorded action is the controller's clean command, its correction from the perturbed state (Laskey et al., 2017). The command the environment actually executed at step t is stored in obs.state.other.action of frame t+1, so replaying action open loop does not reproduce the recorded episodes. Each episode randomizes the toast position (±3 cm in x, ±4 cm in y) and yaw (any heading), the toaster position (±2 cm) and yaw (±8°), the start wrist pose (±5 cm and ±12° per axis) and the relaxed finger pose; 200 of 218 attempts were kept, the other 18 failed the controller's own checks and were discarded.
The recording uses 8 physics substeps per control step (physics_substeps: 8 in meta/env_config.json; DexSuite's default is 2), so build evaluation environments from meta/env_config.json.
The robot is an arm-less floating wrist (a virtual 6-DoF gantry) carrying the right Sharpa hand (22 DoF); the 29-D action is 7 osc_pose_abs_quat values (position offset and orientation relative to the episode-start wrist pose) followed by 22 hand joint position targets in radians. The hand is sharpa_fardexbot, the right Sharpa hand carrying the wrist-camera mount of the real handheld rig (mount plate, quick changer and D405 body, which collide with the scene and show in the front-camera frames); DexSuite ships it under that name. Besides the standard DexSuite keys, every frame carries per-finger contact forces (obs.state.other.finger_force_world, finger_force_norm in N, finger_force_display, finger_force_saturated, finger_contact; thumb, index, middle, ring, pinky) and the controller's phase (obs.state.other.controller_phase: 0 free, 1 approach, 2 in hand) and stage (obs.state.other.controller_stage, named in meta/controller.json). meta/env_config.json rebuilds the environment and meta/controller_episodes/episode_*.json keeps each episode's scene seed, style seed and controller parameters. Depth was rendered during collection but is not included.
- Homepage: https://huggingface.co/dexsuite
- Paper: https://arxiv.org/abs/2610.37799
- License: apache-2.0
Loading
from lerobot.datasets.lerobot_dataset import LeRobotDataset
dataset = LeRobotDataset("dexsuite/put_bread_toaster_floating_sharpa_scripted")
frame = dataset[0] # action, obs.state.*, obs.cameras.front.rgb and obs.cameras.wrist.rgb (3x224x224 float in [0, 1])
Training
DexSuite's robomimic entry point converts the set to HDF5 and trains on it:
python -m scripts.imitation_learning.robomimic.train \
--dataset put_bread_toaster_floating_sharpa_scripted \
--config bc \
--sensors state \
--architecture mlp
--sensors cameras --cameras front trains on the front camera and robot proprio instead, and --config act, act_img or diffusion select the other policy recipes (see scripts/imitation_learning/robomimic/README.md). --dataset dexsuite/put_bread_toaster_floating_sharpa_scripted works as well.
Rollouts build the environment from meta/env_config.json, 8 physics substeps included.
Dataset Structure
{
"codebase_version": "v3.0",
"robot_type": null,
"total_episodes": 200,
"total_frames": 65523,
"total_tasks": 1,
"chunks_size": 1000,
"data_files_size_in_mb": 100,
"video_files_size_in_mb": 200,
"fps": 20,
"splits": {
"train": "0:200"
},
"data_path": "data/chunk-{chunk_index:03d}/file-{file_index:03d}.parquet",
"video_path": "videos/{video_key}/chunk-{chunk_index:03d}/file-{file_index:03d}.mp4",
"features": {
"obs.state.gripper.qpos": {
"dtype": "float32",
"shape": [
22
]
},
"obs.state.gripper.qtorque": {
"dtype": "float32",
"shape": [
22
]
},
"obs.state.gripper.qvel": {
"dtype": "float32",
"shape": [
22
]
},
"obs.state.gripper.tcp_pos": {
"dtype": "float32",
"shape": [
3
]
},
"obs.state.gripper.tcp_quat": {
"dtype": "float32",
"shape": [
4
]
},
"reward": {
"dtype": "float32",
"shape": [
1
]
},
"truncated": {
"dtype": "bool",
"shape": [
1
]
},
"terminated": {
"dtype": "bool",
"shape": [
1
]
},
"success": {
"dtype": "bool",
"shape": [
1
]
},
"failure": {
"dtype": "bool",
"shape": [
1
]
},
"action": {
"dtype": "float32",
"shape": [
29
]
},
"obs.state.manipulator.eef_pos": {
"dtype": "float32",
"shape": [
1,
3
]
},
"obs.state.manipulator.eef_quat": {
"dtype": "float32",
"shape": [
1,
4
]
},
"obs.state.manipulator.qpos": {
"dtype": "float32",
"shape": [
6
]
},
"obs.state.manipulator.qtorque": {
"dtype": "float32",
"shape": [
6
]
},
"obs.state.manipulator.qvel": {
"dtype": "float32",
"shape": [
6
]
},
"obs.state.other.action": {
"dtype": "float32",
"shape": [
29
]
},
"obs.state.other.dial_q": {
"dtype": "float32",
"shape": [
1
]
},
"obs.state.other.last_action": {
"dtype": "float32",
"shape": [
29
]
},
"obs.state.other.lever_frac": {
"dtype": "float32",
"shape": [
1
]
},
"obs.state.other.lever_q": {
"dtype": "float32",
"shape": [
1
]
},
"obs.state.other.toast_aabb_hi": {
"dtype": "float32",
"shape": [
3
]
},
"obs.state.other.toast_aabb_lo": {
"dtype": "float32",
"shape": [
3
]
},
"obs.state.other.toast_ang": {
"dtype": "float32",
"shape": [
3
]
},
"obs.state.other.toast_in_slot": {
"dtype": "float32",
"shape": [
1
]
},
"obs.state.other.toast_in_toaster_frame": {
"dtype": "float32",
"shape": [
3
]
},
"obs.state.other.toast_pos": {
"dtype": "float32",
"shape": [
3
]
},
"obs.state.other.toast_quat": {
"dtype": "float32",
"shape": [
4
]
},
"obs.state.other.toast_vel": {
"dtype": "float32",
"shape": [
3
]
},
"obs.state.other.toaster_pos": {
"dtype": "float32",
"shape": [
3
]
},
"obs.state.other.toaster_quat": {
"dtype": "float32",
"shape": [
4
]
},
"obs.cameras.front.rgb": {
"dtype": "video",
"shape": [
224,
224,
3
],
"info": {
"video.height": 224,
"video.width": 224,
"video.codec": "av1",
"video.pix_fmt": "yuv420p",
"video.is_depth_map": false,
"video.fps": 20,
"video.channels": 3,
"has_audio": false
}
},
"obs.cameras.wrist.rgb": {
"dtype": "video",
"shape": [
224,
224,
3
],
"info": {
"video.height": 224,
"video.width": 224,
"video.codec": "av1",
"video.pix_fmt": "yuv420p",
"video.is_depth_map": false,
"video.fps": 20,
"video.channels": 3,
"has_audio": false
}
},
"obs.state.other.finger_force_world": {
"dtype": "float32",
"shape": [
5,
3
]
},
"obs.state.other.finger_force_norm": {
"dtype": "float32",
"shape": [
5
]
},
"obs.state.other.finger_force_display": {
"dtype": "float32",
"shape": [
5
]
},
"obs.state.other.finger_force_saturated": {
"dtype": "bool",
"shape": [
5
]
},
"obs.state.other.finger_contact": {
"dtype": "bool",
"shape": [
5
]
},
"obs.state.other.controller_phase": {
"dtype": "int64",
"shape": [
1
]
},
"obs.state.other.controller_stage": {
"dtype": "int64",
"shape": [
1
]
},
"timestamp": {
"dtype": "float32",
"shape": [
1
],
"names": null
},
"frame_index": {
"dtype": "int64",
"shape": [
1
],
"names": null
},
"episode_index": {
"dtype": "int64",
"shape": [
1
],
"names": null
},
"index": {
"dtype": "int64",
"shape": [
1
],
"names": null
},
"task_index": {
"dtype": "int64",
"shape": [
1
],
"names": null
}
},
"task": "put_bread_toaster",
"options": null
}
Citation
BibTeX:
[More Information Needed]
- Downloads last month
- 71