NVIDIA / NVIDIA/cosmos-framework
Why does the conditioning image only go through VAE and not ViT
Open
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 535
- Forks
- 148
- Avg merge
- 13h 37m
- Merged PRs (30d)
- 35
Description
I noticed that in Generator Mode, the conditioning image is only passed through the VAE and is not fed through the ViT for high-level semantic understanding. I’m curious about the reasoning behind this design choice. Were any ablation studies conducted to validate it? Thank you!
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
No file or test is identified in the issue. Inspect the Generator Mode conditioning path and any existing documentation or ablation records for the VAE and ViT choices; done means documenting the design rationale and reporting whether ablation studies were conducted.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- machine-learning
- Issue type
- Documentation
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100