Re-run the precision-invariance experiment under matched stopping (current numbers are uncontrolled)

Open
#209 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
5/5
Estimated time
Over a week
Newbie friendliness
35/100
Issue type
Bug
Clarity
Mostly clear
Activity status
Quiet
Tech stack
fortran, python, pytorch

Research direction

Start with .context/issue-90/ksweep_findings.md:56, docs/guides/validation.md:200 and 278-284, and the changes from #207 before rerunning the experiment. Use identical seeds and initialization, matched stopping, separate precision and implementation contrasts, and a k sweep while recording log-likelihood and correlation. Done means the controlled results are reported and the float32 guidance is re-derived from them.

Written by the indexing model from the issue text.

Description

Surfaced via #206, where 0.971 was read as the cost of float32. Investigating that reading showed the number cannot support any float32 claim, in either direction: the comparison is not controlled.

The defect is in the experiment, not the wording

.context/issue-90/ksweep_findings.md:56 records the cause:

at k=152 the f32 run hit the natural-gradient lrate floor (1e-12) at iter 1735 and stopped early, landing on a marginally different optimum than the full-2000-iter f64 runs.

So the k=152 rows in docs/guides/validation.md:278-284 compare a 1735-iteration f32 run against 2000-iteration f64 runs:

Pair (at k=152) |corr| what it actually varies
native-Fortran f64 vs PyTorch-CUDA f64 0.995 implementation
native-Fortran f64 vs PyTorch-CUDA f32 0.971 implementation + precision + 265 fewer iterations
PyTorch-CUDA f64 vs PyTorch-CUDA f32 0.979 precision + 265 fewer iterations

Precision and iteration count are confounded. Neither 0.971 nor 0.979 isolates anything, so no amount of relabelling makes them mean what the surrounding prose says they mean. The published guide currently attributes the residual to "the float32 path" while the controlled contrast that would establish it was never run.

This also sits badly against docs/guides/validation.md:200, which calls k=30 "the definitive float32 == float64 result" at 1.000. Both arms completed 2000 iterations there, so that one IS controlled, but the two sections then appear to contradict each other with no stated reason.

What a correct experiment looks like

  • Matched stopping across arms. Either pin all arms to an identical iteration count, or record and report the per-arm stop iteration and reason alongside every number.
  • Identical seed and initialization across arms, so precision is the only variable in the precision contrast.
  • Separate the contrasts rather than mixing them in one table: precision alone (same implementation, same device, f32 vs f64) and implementation alone (Fortran f64 vs torch f64).
  • Sweep k, since the effect is k-dependent and a single k cannot distinguish "precision matters more on larger data" from "the f32 arm stopped early on larger data".
  • Report log-likelihood as well as matched component correlation. LL is the objective being optimized; correlation is a downstream proxy and is degenerate on under-determined components.
  • Re-derive the guide's float32 guidance from whatever the controlled numbers show, rather than editing the prose around the existing ones.

Sequencing: blocked on #207

#207 adds the missing Fortran convergence criteria (min_dll/maxincs, min_nd/use_grad_norm) to AMICATorchNG. The early stop that confounded this experiment was lrate_floor firing, so #207 changes stopping behaviour directly. Re-running before it lands would produce numbers that are stale on arrival.

Scope note

docs/guides/validation.md is the reproduction reference cited by the JOSS paper (openjournals/joss-reviews#11093). The paper itself claims "single-precision runs agree with double precision to four or five significant digits", which is a log-likelihood claim rather than a correlation claim and is not the same statement as these rows; check whether it traces to a controlled run before the paper is finalized.

Dominant language
Python
Stars
8
Forks
3
Avg merge
5h 37m
Merged PRs (30d)
36

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from sccn/pAMICA

All issues in sccn/pAMICA

Similar issues

More Python issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.