Single-Object Multi-Stage Manipulation Task

Full Dataset (545Gb)

Multi-Object Multi-Stage Manipulation Task

Assembly 1 (86Gb)

Assembly 2 (77Gb)

Assembly 3 (70Gb)

Data Format

Each zip file contains a folder of trajectories. Each trajectory is saved as a .npy file. Each .npy file condtains a dictionary with the following keys value pairs:

• obs/side_1 a (N, 256, 256, 3) numpy array of RGB images from the side camera 1 saved in BGR format
• obs/side_2 a (N, 256, 256, 3) numpy array of RGB images from the side camera 2 saved in BGR format
• obs/wrist_1 a (N, 256, 256, 3) numpy array of RGB images from the wrist camera 1 saved in BGR format
• obs/wrist_2 a (N, 256, 256, 3) numpy array of RGB images from the wrist camera 2 saved in BGR format
• obs/side_1_depth a (N, 256, 256) numpy array of depth images from the side camera 1
• obs/side_2_depth a (N, 256, 256) numpy array of depth images from the side camera 2
• obs/wrist_1_depth a (N, 256, 256) numpy array of depth images from the wrist camera 1
• obs/wrist_2_depth a (N, 256, 256) numpy array of depth images from the wrist camera 2
• obs/tcp_pose a (N, 7) numpy array of the end effector pose in the robot's base frame (XYZ, Quaternion)
• obs/tcp_vel a (N, 6) numpy array of the end effector velocity in the robot's base frame (XYZ, RPY)
• obs/tcp_force a (N, 3) numpy array of the end-effector force in the end-effector frame (XYZ)
• obs/tcp_torque a (N, 3) numpy array of the end-effector torque in the end-effector frame (RPY)
• obs/q a (N, 7) numpy array of the joint positions
• obs/dq a (N, 7) numpy array of the joint velocities
• obs/jacobian a (N, 6, 7) numpy array of the robot jacobian
• obs/gripper_pose a (N, ) numpy array of indicating the binary state of the gripper (0=open, 1=closed)
• action a (N, 7) numpy array of the commanded carteasian action (XYZ, RPY, gripper)
• primitive a (N, ) numpy array of strings indicating the primitive associated with the current timestep
• object_id (Multi-Object only) a (N, ) numpy array of integers indicating the ID the object being manipulated in the current trajectory
• object_info (Single-Object only) a dictionary contain information of the object being manipulated in the current trajectory with the following keys-value pairs
• length length of the object (Short, Long)
• size cross-sectional size of the object (Small, Medium, Large)
• shape shape ID of the object according to reference sheet
• color color ID of the object according to reference sheet
• angle initial pose of the object indicating how it should be grasped and reoriented (horizontal, vertical)
• distractor indicator for whether there are distractor objects (yes, no)

File Naming

The Single-Object Dataset trajectory files are named as following:

(insert_only_){shape}_{size}_{length}_{color}_{angle}_{distractor}_{trajectory_id}.npy

The Multi-Object Dataset trajectory files are named as following:

trajectory_{object_id}_{trajectory_id}.npy

Using Point Clouds

The camera intrinsics are provided in the files below. They can be used to convert the depth images to point cloud for your application.

• Side Camera 1 Intrinsics
• Side Camera 2 Intrinsics
• Wrist Camera 1 Intrinsics
• Wrist Camera 2 Intrinsics