Assembly 2 (77Gb)

Assembly 3 (70Gb)

`.npy`

file. Each
`.npy`

file
condtains a dictionary with the following keys value pairs:
`obs/side_1`

a (N, 256, 256, 3) numpy array of RGB images from the side camera 1 saved in BGR format`obs/side_2`

a (N, 256, 256, 3) numpy array of RGB images from the side camera 2 saved in BGR format`obs/wrist_1`

a (N, 256, 256, 3) numpy array of RGB images from the wrist camera 1 saved in BGR format`obs/wrist_2`

a (N, 256, 256, 3) numpy array of RGB images from the wrist camera 2 saved in BGR format`obs/side_1_depth`

a (N, 256, 256) numpy array of depth images from the side camera 1`obs/side_2_depth`

a (N, 256, 256) numpy array of depth images from the side camera 2`obs/wrist_1_depth`

a (N, 256, 256) numpy array of depth images from the wrist camera 1`obs/wrist_2_depth`

a (N, 256, 256) numpy array of depth images from the wrist camera 2`obs/tcp_pose`

a (N, 7) numpy array of the end effector pose in the robot's base frame (XYZ, Quaternion)`obs/tcp_vel`

a (N, 6) numpy array of the end effector velocity in the robot's base frame (XYZ, RPY)`obs/tcp_force`

a (N, 3) numpy array of the end-effector force in the end-effector frame (XYZ)`obs/tcp_torque`

a (N, 3) numpy array of the end-effector torque in the end-effector frame (RPY)`obs/q`

a (N, 7) numpy array of the joint positions`obs/dq`

a (N, 7) numpy array of the joint velocities`obs/jacobian`

a (N, 6, 7) numpy array of the robot jacobian`obs/gripper_pose`

a (N, ) numpy array of indicating the binary state of the gripper (0=open, 1=closed)`action`

a (N, 7) numpy array of the commanded carteasian action (XYZ, RPY, gripper)`primitive`

a (N, ) numpy array of strings indicating the primitive associated with the current timestep`object_id (Multi-Object only)`

a (N, ) numpy array of integers indicating the ID the object being manipulated in the current trajectory`object_info (Single-Object only)`

a dictionary contain information of the object being manipulated in the current trajectory with the following keys-value pairs`length`

length of the object (**S**hort,**L**ong)`size`

cross-sectional size of the object (**S**mall,**M**edium,**L**arge)`shape`

shape ID of the object according to reference sheet`color`

color ID of the object according to reference sheet`angle`

initial pose of the object indicating how it should be grasped and reoriented (**horizontal**,**vertical**)`distractor`

indicator for whether there are distractor objects (**y**es,**n**o)

`(insert_only_){shape}_{size}_{length}_{color}_{angle}_{distractor}_{trajectory_id}.npy`

The Multi-Object Dataset trajectory files are named as following:

`trajectory_{object_id}_{trajectory_id}.npy`

- Side Camera 1 Intrinsics
- Side Camera 2 Intrinsics
- Wrist Camera 1 Intrinsics
- Wrist Camera 2 Intrinsics