microsoft / microsoft/TRELLIS.2
Multi-GPU training crashes with flex_gemm
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 11.3k
- Forks
- 1.4k
- PR merge metrics
- No merged PRs in 30d
Description
Trying to train on 2x H200 but it crashes right after start with “terminate called without an active exception” repeating over and over.
NCCL connects fine, both GPUs show Init COMPLETE, then crashes on first forward pass.
Single GPU works, ran 2000 steps. But also crashed on sampling with “illegal memory access”.
Tried different ports, xformers, nothing helped. spconv not included in setup.sh so couldn’t test that.
Is multi-GPU actually tested?
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
No source file or test is named. Start by reproducing the first forward pass on two H200 GPUs and the sampling failure on one GPU, then inspect setup.sh and the flex_gemm training and sampling entry points. Done means the reported crashes are isolated and a verified fix or clear reproduction is documented.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- distributed-systems, machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100