microsoft / microsoft/TRELLIS.2

Hard-thresholded logit decisions cause scattered surface holes on non-CUDA platforms (data + analysis from a working ROCm port)

Open
#169 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
11.3k
Forks
1.4k
PR merge metrics
No merged PRs in 30d

Description

## Summary

We have TRELLIS.2 image→3D running end-to-end on AMD ROCm (gfx1151,
Strix Halo; related: #74). After fixing platform bugs (CuMesh
`cudaMemcpy2D` failing above ~1M rows on HIP — fix PR'd to CuMesh:
https://github.com/JeffreyXiang/CuMesh/pull/35), output quality is close to reference but not equal: meshes carry
tens of thousands of micro boundary defects (missing single triangles,
missing vertex fans) that NVIDIA outputs do not have.

Reference comparison: official HF Space (same input image, seed=0, 512)
produces a fully watertight mesh after welding UV seams — 0 boundary
edges. Our ROCm mesh: ~220K boundary edges before fill_holes, ~95% of
visible artifacts removed by fill_holes, but a floor remains.

## What it is NOT (all verified directly)

- Not sparse-conv kernels: FlexGEMM submanifold conv unit-tested vs
  brute-force reference at 69K and 1.8M voxels — max err 1e-3 (fp16
  rounding), zero outliers, both algorithms.
- Not the dual-grid converter: two independent implementations (CUDA/HIP
  kernels and a pure-PyTorch port) produce the same defect pattern.
- Not NaN/Inf: decoder outputs at the extraction boundary instrumented —
  zero NaN/Inf.
- Not fp16 range: bf16 mixed-precision decoder produces the same pattern.
- Not attention: the FDG VAE decoder contains no attention layers.

## Proposed mechanism

The decoder makes hard binary decisions on learned logits at multiple
points: `to_subdiv logits > 0` (child voxel creation, per upsample level)
and intersection logits `> 0` (quad emission at extraction). Logits within
~±1e-3 of zero — ordinary cross-platform fp16 accumulation differences —
flip these decisions per-cell, producing scattered missing cells/quads.
Consistent with this: globally biasing the decisions positive
(bias += 0.1) halves the defect count but visibly changes the generation
(smoother, loses fine detail), while applying the same shift only at
decode time has no effect — the sensitivity is distributed across the
sampling/upsample chain, not localized at extraction.

## Suggestion

If portability across accelerators is a goal, the hard `> 0` thresholds
are the fragile point. Options that would help downstream ports:
- a documented/configurable decision margin,
- training-time margin regularization on these logits (push them away
  from 0), or
- an official statement of the expected defect rate + the intended
  fill_holes responsibility, so ports know what "correct" looks like.

full package and input and output intermediate glb files available at:
https://github.com/ron-ytsma/trellis2-rocm-gfx1151/releases/tag/1.0.0

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the FDG VAE decoder's hard to_subdiv logits > 0 and intersection-logit decisions, then compare the ROCm and NVIDIA outputs using the provided release artifacts. Reproduce the boundary-edge counts and inspect the sampling, upsample, extraction, and fill_holes stages; done requires an agreed portability behavior or documented expected defect rate, since the issue presents several possible directions rather than a selected fix.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, pytorch
Domain
computer-graphics, machine-learning
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.