Eval mode 2 does part-wise Rel-L2 averaging, not joint full-mesh Rel-L2

Open
#5 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
55/100
Issue type
Bug
Clarity
Mostly clear
Activity status
Quiet
Tech stack
python
Domain
machine-learning

Research direction

Read dataset/dataset_drivaerml_surface_numpy_chunk.py and the eval paths in train_surface.py, especially train_surface.test and test_decoupled_inference. Trace how mode 2 batches and aggregates Rel-L2, then verify whether published values use part-wise averaging or full-mesh accumulation. Done means the intended Eq. 7 evaluation behavior is established and the relevant evaluation path reports one ratio per mesh.

Written by the indexing model from the issue text.

Description

Observation

Eval mode 2 (--eval 2: decoupled inference / physical state caching + full mesh decoding) does not appear to do joint prediction + Rel-L2 over all cells in a mesh. It does piecewise predictions over parts, then averages per-part Rel-L2s.

That conflicts with Appendix A.3 Eq. 7 (Rel-L2 over the full mesh of (N) cells).

Evidence

Test-mode dataloader yields one part at a time, not the entire mesh:

https://github.com/thuml/Transolver-3/blob/main/dataset/dataset_drivaerml_surface_numpy_chunk.py

  • train=False: length is #runs × #parts
  • __getitem__ returns a single stride-part

train_surface.test and test_decoupled_inference Stage 2 then compute Rel-L2 on that part and np.mean over batches:

https://github.com/thuml/Transolver-3/blob/main/train_surface.py

So reported L2RE is mean_parts(RelL2(part)), not joint RelL2(entire mesh).

Mode 2 Stage 1 builds a global cache, but Stage 2 still decodes/evaluates part-by-part with that averaging.

Question

Were published surface Rel-L2 numbers computed this way, or via true full-mesh accumulation? If Eq. 7 is intended, eval should sum error/target energies across parts (per run) and take one ratio.

Dominant language
Python
Stars
34
Forks
6
No merged PRs in 30d

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from thuml/Transolver-3

All issues in thuml/Transolver-3

Similar issues

More Python issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.