NVIDIA / NVIDIA/cosmos-framework

CPU-side ColorJitter is a dataloader bottleneck — could the GPU path be restored?

Open
#174 4 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
535
Forks
148
Avg merge
13h 37m
Merged PRs (30d)
35

Description

DROID action SFT currently runs ColorJitter inside the dataloader workers, which is
computationally expensive.

  1. In principle: ColorJitter is a random augmentation to begin with, so moving it to
    the GPU should only introduce CPU/CUDA kernel-level numerical differences, without
    changing the training semantics — is that right?

  2. In practice: ColorJitter and the tail after it can be moved out of the dataloader
    worker and run in the main process on the GPU. The pipeline stage and the ordering stay exactly
    the same; only the execution device changes. The worker also passes along its RNG
    state, which keeps the final difference limited to CPU/CUDA kernel rounding.
    The cost is that worker-to-main-process traffic doubles, since the tensor returned
    by the worker goes from [T, C, 3H/2, W] to [3T, C, H, W].

Does this approach look viable?

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by locating the DROID action SFT dataloader and its ColorJitter pipeline, then trace where the tail stage and worker RNG state are handled. Compare the proposed worker-to-main-process transfer with the current ordering; done means the GPU path is shown viable without semantic changes and its traffic or performance cost is measured.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
machine-learning, performance
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.