microsoft / microsoft/TRELLIS.2
Hard-thresholded logit decisions cause scattered surface holes on non-CUDA platforms (data + analysis from a working ROCm port)
Open
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 11.3k
- Forks
- 1.4k
- PR merge metrics
- No merged PRs in 30d
Description
## Summary
We have TRELLIS.2 image→3D running end-to-end on AMD ROCm (gfx1151,
Strix Halo; related: #74). After fixing platform bugs (CuMesh
`cudaMemcpy2D` failing above ~1M rows on HIP — fix PR'd to CuMesh:
https://github.com/JeffreyXiang/CuMesh/pull/35), output quality is close to reference but not equal: meshes carry
tens of thousands of micro boundary defects (missing single triangles,
missing vertex fans) that NVIDIA outputs do not have.
Reference comparison: official HF Space (same input image, seed=0, 512)
produces a fully watertight mesh after welding UV seams — 0 boundary
edges. Our ROCm mesh: ~220K boundary edges before fill_holes, ~95% of
visible artifacts removed by fill_holes, but a floor remains.
## What it is NOT (all verified directly)
- Not sparse-conv kernels: FlexGEMM submanifold conv unit-tested vs
brute-force reference at 69K and 1.8M voxels — max err 1e-3 (fp16
rounding), zero outliers, both algorithms.
- Not the dual-grid converter: two independent implementations (CUDA/HIP
kernels and a pure-PyTorch port) produce the same defect pattern.
- Not NaN/Inf: decoder outputs at the extraction boundary instrumented —
zero NaN/Inf.
- Not fp16 range: bf16 mixed-precision decoder produces the same pattern.
- Not attention: the FDG VAE decoder contains no attention layers.
## Proposed mechanism
The decoder makes hard binary decisions on learned logits at multiple
points: `to_subdiv logits > 0` (child voxel creation, per upsample level)
and intersection logits `> 0` (quad emission at extraction). Logits within
~±1e-3 of zero — ordinary cross-platform fp16 accumulation differences —
flip these decisions per-cell, producing scattered missing cells/quads.
Consistent with this: globally biasing the decisions positive
(bias += 0.1) halves the defect count but visibly changes the generation
(smoother, loses fine detail), while applying the same shift only at
decode time has no effect — the sensitivity is distributed across the
sampling/upsample chain, not localized at extraction.
## Suggestion
If portability across accelerators is a goal, the hard `> 0` thresholds
are the fragile point. Options that would help downstream ports:
- a documented/configurable decision margin,
- training-time margin regularization on these logits (push them away
from 0), or
- an official statement of the expected defect rate + the intended
fill_holes responsibility, so ports know what "correct" looks like.
full package and input and output intermediate glb files available at:
https://github.com/ron-ytsma/trellis2-rocm-gfx1151/releases/tag/1.0.0
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the FDG VAE decoder's hard to_subdiv logits > 0 and intersection-logit decisions, then compare the ROCm and NVIDIA outputs using the provided release artifacts. Reproduce the boundary-edge counts and inspect the sampling, upsample, extraction, and fill_holes stages; done requires an agreed portability behavior or documented expected defect rate, since the issue presents several possible directions rather than a selected fix.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, pytorch
- Domain
- computer-graphics, machine-learning
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100