microsoft / microsoft/TRELLIS.2
Abnormal VRAM Growth During Batch Generation
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 11.3k
- Forks
- 1.4k
- PR merge metrics
- No merged PRs in 30d
Description
Hi, thank you for presenting such an impressive work! However, when I modified the provided single-sample generation script to support batched generation, I encountered an issue of abnormal VRAM growth. Specifically, even after explicitly invoking memory-release operations such as torch.cuda.empty_cache(), the GPU memory usage still exhibits a clear linear increase over time.
As shown in the figure below, I performed more than 300 times of generation and logged the GPU memory consumption right after DiT sampling and after o-voxel decoding in each generation and both of them show a consistent linear upward trend.
Do you have any insights or suggestions regarding this unusual behavior? Looking forward your early reply
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reproducing the batched-generation modification of the provided single-sample generation script and record memory after DiT sampling and o-voxel decoding. Trace those stages and the existing torch.cuda.empty_cache() calls; done means repeated generation no longer shows a consistent linear VRAM increase.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100