ByteDance-Seed / ByteDance-Seed/VideoWorld

Question about GPU setup for dLDM stage-2 training

Open
#15 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
801
Forks
40
PR merge metrics
No merged PRs in 30d

Description

Hi, thanks for releasing VideoWorld 2!

I am trying to reproduce the dLDM training pipeline, and kept the stage-2 setting same as your released config:

- `predict2_video2world_ctrl_ldm_fsdp_2b_480p_16fps`
- `train_architecture="base"`
- `finetune_base_model=True`
- `use_cross_embedding=True`
- `batch_size=1` per GPU
- `num_frames=93`
- `video_size=(480, 832)`
- `fsdp_shard_size=8`
- 8 GPUs total

From the logs, stage 2 is training roughly:
- DiT trainable parameters: `2.20B`
- dLDM trainable parameters: `0.14B`

So this is effectively joint training of:
- full DiT
- full control branch
- dLDM

I can confirm that FSDP is enabled (`Distributed parallelism mode: fsdp`, `Using FSDP with shard size 8`), but I still run into OOM issues on 8 GPUs with 48 GB VRAM each.

Because of this, I would like to ask:

1. What exact GPU setup did you use for dLDM stage-2 training?
- number of GPUs
- VRAM per GPU
- GPU model

2. For the released training recipe, was stage 2 expected to fit on 8 x 48 GB GPUs?

3. Did you use any additional memory-saving settings beyond the released config?
For example:
- activation checkpointing changes
- different batch size / grad accumulation
- context parallelism
- different FSDP settings
- partial freezing instead of full base-model finetuning

4. If stage 2 in your internal training used a different setup than the public config, could you share the practical hardware recommendation for reproducing it?

Any clarification would be very helpful for reproduction. Thanks.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.