Synthetic Data Generation Pipeline

UPDATE on 06/01/2026: We rescope the work and focus on imitation learning in IsaacSim and impact of human demonstration in the training.

Goal: Create a pipeline that takes a few episodes of OpenArm manipulation and generates over a few K episodes.

Main Tasks:

• Obtain Doga's OpenARM robot manipulation dataset as human-guided episode
• (RESCOPED) Try tools like DexMimicGen and design the pipeline to Create a scene with target objects and OpenArm robot
• (NEW) Use OpenArm leader arm or keyboard to operate OpenArm in the Isaac scene and collect human demonstrations
• (NEW) Generate additional demonstrations from human demonstration using IsaacLab Mimic
• (NEW) Train AI and evaluate performance (with variations in the amount of demonstrations and training parameters)
• (DROPPED) Generate few K episodes for a couple of robot tasks (e.g., pick-and-place cubic and open closet door)
• Replay episodes in Isaac Sim

Optional Tasks:

• (NEW) Compare performance between Imitation learning and ACT learning, understanding what it means to obtain human demonstration for autonomous robot manipulation.
• Augment other sensor data (e.g., tactile) in generated synthetic dataset
• Analyze # of episodes VS training accuracy (if we can finetune a VLA model with this dataset)

Success Criteria:

• End-to-end workflow document showing an example workflow from putting episodes for a task and getting x1000 dataset
• (DESCOPED) Getting thousands of episode for the two robot tasks
• AI models trained on the target task (putting plastic tubes into centrifuge machine)