Re-run the precision-invariance experiment under matched stopping (current numbers are uncontrolled)
Nobody has claimed this yet.
Assessment
- Difficulty
- 5/5
- Estimated time
- Over a week
- Newbie friendliness
- 35/100
- Issue type
- Bug
- Clarity
- Mostly clear
- Activity status
- Quiet
- Tech stack
- fortran, python, pytorch
- Domain
- documentation, machine-learning
Research direction
Start with .context/issue-90/ksweep_findings.md:56, docs/guides/validation.md:200 and 278-284, and the changes from #207 before rerunning the experiment. Use identical seeds and initialization, matched stopping, separate precision and implementation contrasts, and a k sweep while recording log-likelihood and correlation. Done means the controlled results are reported and the float32 guidance is re-derived from them.
Written by the indexing model from the issue text.
Description
Surfaced via #206, where 0.971 was read as the cost of float32. Investigating that reading showed the number cannot support any float32 claim, in either direction: the comparison is not controlled.
The defect is in the experiment, not the wording
.context/issue-90/ksweep_findings.md:56 records the cause:
at k=152 the f32 run hit the natural-gradient lrate floor (1e-12) at iter 1735 and stopped early, landing on a marginally different optimum than the full-2000-iter f64 runs.
So the k=152 rows in docs/guides/validation.md:278-284 compare a 1735-iteration f32 run against 2000-iteration f64 runs:
| Pair (at k=152) | |corr| | what it actually varies |
|---|---|---|
| native-Fortran f64 vs PyTorch-CUDA f64 | 0.995 | implementation |
| native-Fortran f64 vs PyTorch-CUDA f32 | 0.971 | implementation + precision + 265 fewer iterations |
| PyTorch-CUDA f64 vs PyTorch-CUDA f32 | 0.979 | precision + 265 fewer iterations |
Precision and iteration count are confounded. Neither 0.971 nor 0.979 isolates anything, so no amount of relabelling makes them mean what the surrounding prose says they mean. The published guide currently attributes the residual to "the float32 path" while the controlled contrast that would establish it was never run.
This also sits badly against docs/guides/validation.md:200, which calls k=30 "the definitive float32 == float64 result" at 1.000. Both arms completed 2000 iterations there, so that one IS controlled, but the two sections then appear to contradict each other with no stated reason.
What a correct experiment looks like
- Matched stopping across arms. Either pin all arms to an identical iteration count, or record and report the per-arm stop iteration and reason alongside every number.
- Identical seed and initialization across arms, so precision is the only variable in the precision contrast.
- Separate the contrasts rather than mixing them in one table: precision alone (same implementation, same device, f32 vs f64) and implementation alone (Fortran f64 vs torch f64).
- Sweep k, since the effect is k-dependent and a single k cannot distinguish "precision matters more on larger data" from "the f32 arm stopped early on larger data".
- Report log-likelihood as well as matched component correlation. LL is the objective being optimized; correlation is a downstream proxy and is degenerate on under-determined components.
- Re-derive the guide's float32 guidance from whatever the controlled numbers show, rather than editing the prose around the existing ones.
Sequencing: blocked on #207
#207 adds the missing Fortran convergence criteria (min_dll/maxincs, min_nd/use_grad_norm) to AMICATorchNG. The early stop that confounded this experiment was lrate_floor firing, so #207 changes stopping behaviour directly. Re-running before it lands would produce numbers that are stale on arrival.
Scope note
docs/guides/validation.md is the reproduction reference cited by the JOSS paper (openjournals/joss-reviews#11093). The paper itself claims "single-precision runs agree with double precision to four or five significant digits", which is a log-likelihood claim rather than a correlation claim and is not the same statement as these rows; check whether it traces to a controlled run before the paper is finalized.
- Dominant language
- Python
- Stars
- 8
- Forks
- 3
- Avg merge
- 5h 37m
- Merged PRs (30d)
- 36
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from sccn/pAMICA
-
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
-
Difficulty 4/5 3-5 days Newbie friendliness 48/100
-
Difficulty 3/5 1-2 days Newbie friendliness 68/100
-
Difficulty 5/5 Over a week Newbie friendliness 35/100
-
Difficulty 5/5 Over a week Newbie friendliness 35/100
Similar issues
-
Difficulty 2/5 1-3 hours Newbie friendliness 74/100
bancolombia/sentinel#23 ·
-
test md OpenCI
Difficulty 2/5 1-3 hours Newbie friendliness 74/100
-
integration:quickjs org:external priority:backlog topic:code-interpreter topic:middleware type:feature
Difficulty 2/5 1-3 hours Newbie friendliness 74/100
langchain-ai/deepagents#6450 ·
-
bug client
Difficulty 2/5 1-3 hours Newbie friendliness 88/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 74/100