TrainerRank reusable-cache admission can leave insufficient physical CUDA memory for backward library allocations

Current status — September 18, 2026, late evening Mountain: partially addressed; still open.

PR #900 is merged. Current held PR #888 is at f557bc4e60f7b38bead1cc0efd7d655e2bc7df69; its low-physical-headroom backward handoff now uses the same measured cache-release budget, rather than an unconditional release rule. Three exact-head source reviews and CI, including two-H200 checks, are clear. Significant behavioral adoption still needs Brad's decision.

A full-model candidate run completed three real optimizer updates (18 backwards), followed by a separate pressure diagnostic with one automatic handoff release and one successful backward, with no optimizer update during that pressure diagnostic. It then successfully published the grouped generator/auxiliary checkpoint. The original run nevertheless exited 1: after trainer close returned, its actor-absence check timed out before fresh reload began. This failure is retained and is not classified as a backend or zombie-process cause. Exact owned resources were independently retired afterward.

A distinct load-only continuation is now running on one free H200: art848ws-07a21878, launched at 11:04:39 PM Mountain. It uses protocol-8 Caladan with the original candidate ART source, loads the exact saved generator:v1/auxiliary:v1 group from art848ws-b140393f, compares complete canonical optimizer/custom state, then attempts exactly one update and real cleanup. Prior training, pressure and remote grouped publication are not repeated. Its actual worker-image and restored-helper CPU checks passed; native saved-state equality and successful completion are not yet claimed.

The observed release cost is not a total-training 5% overhead bound. Repeat-policy instrumentation itself contributes to measured work/cost, and process-wide cache effects remain relevant. A separate source proposal can measure two pressure episodes and intervening updates, but is not yet run and cannot by itself establish representative throughput or arbitrary backward safety. Keep this issue open separately from #848 and numerical variability #902.

Evidence: /var/tmp/art888-load-only-candidate-image-20260919-root/cpu-root-acceptance.json (041f584e), exact native authority gpu-root-admission-v2.json (d4fb5fd6) and gpu-live-identities.json (e16daabc). Original failed producer evidence remains /var/tmp/art888-native-sequence-20260919-backward-n2_crwep/operation/. No merge or deployment occurred in this follow-up.

---

Earlier status/history, preserved; current statements above supersede old head and pending-candidate descriptions:

Current status — September 17, 2026: partially addressed; still open.

Brad approved PR #900, merged as c009557e2f2488a83551d29c3253b708edef322c. Admission now uses physical free CUDA memory and can release unused allocator cache under a measured cost budget. The scoped two-H200 qualification and source/CI checks passed before merge. This does not establish full-training overhead, sufficient memory throughout arbitrary backwards, or adoption by every Caladan experiment/image.

The earlier PR #888 is a separate, still-held proposal to release cache at a low-headroom handoff to backward. It is not adopted by #900, and its process-wide effects and throughput tradeoffs remain relevant. It should not be merged or relabeled as superseded without reassessing that remaining boundary against the current admission policy.

Active work with #848: qualify the current code on the retained workload and resolve underestimated backward demand. Keep the library-allocation problem separate from the repeated-gradient variability tracked by #902. All recent completed diagnostic resources are cleaned up. Representative end-to-end overhead and broader backward-memory safety remain unqualified.

The historical September 12 status below is preserved; its description of the then-current candidate is superseded by this update.

---

September 12 update: the earlier statement that conditional/full-step and multi-device behavior are entirely unqualified is superseded by bounded native evidence. Three complete updates on one H200 exercised four actual conditional releases; a separate two-H200 run verified that selected-device release also trims the other device's unused cache while preserving the checked live values/gradients and caller device/streams. All resources were independently retired. Exact retained evidence and limits: /home/brad/.local/share/schulman/art870-two-device-gap-inventory-20260912-capture/REPORT.md.

ART #888 remains OPEN/DRAFT at1763a46ac9b4f465e38440baf8cc3e7922da69a3, source-correctness reviews and CI green, adoption HELD. The 3% trigger is best effort, not a guaranteed reserve. Process-wide cache retirement/waits are significant behavior requiring Brad's decision. Representative throughput/refill costs, arbitrary concurrent users, and broader distributed/negative cases remain unqualified. Do not repeat the completed two-device witness merely to restate the already observed process-wide effect.

---

Earlier record (preserved; status above is current):

Status reconciliation — September 11, 2026 (Schulman)

Confirmed allocation failure; local mitigation only. Unconditional cache release let the pinned first-backward diagnostic finish, but the proposed conditional trigger, complete optimizer behavior and multi-device side effects remain unqualified. torch.cuda.empty_cache can trim caches across devices in the process; a selected-device context does not isolate that effect.

Planned lane: Schulman and subagents, queued with #848/#256. Complete the conditional/multi-device and full-step qualification before proposing a generic fix. The held local patch is not merged, does not modify art.megatron, and is not covered by a no-significant-behavior assumption.

---

Historical report (preserved):

Type: confirmed backward allocation failure with a controlled experiment-local mitigation; generic policy remains unresolved.
Agent owner: Schulman, related to #848.

A rank-1 Qwen3.6-35B-A3B eight-pair dynamics batch on one H200 passed its forwards, then failed in micro_batch_loss.backward with CUBLAS_STATUS_ALLOC_FAILED from cublasCreate(handle). GeneratorExit was secondary iterator teardown. The exact internal cuBLAS request size was not measured.

At the first gradient admission, live allocated/reserved memory was about 74.662/147.394 decimal GB with only 0.034734 GB physically free. The planner treated reusable allocator slack as available; its 3% capacity subtraction did not physically reserve CUDA memory for external library allocations. At the caller boundary, an unconditional torch.cuda.empty_cache() released 42.754638 GB without changing live allocation. The matched release-only replay completed all five backward waves and the optimizer; the observe-only replay failed. Pre-intervention plans/budgets and corresponding forward counters matched exactly.

The qualified mitigation is experiment-local cache release at the start of every gradient caller, after receiving its microbatch. It is not equivalent to a conditional 3% policy, a pre-yield trainer hook, or handle warmup. Later nonzero-generator-gradient updates also succeeded, but larger traces and full validation remain unqualified. Cold forward estimate underprediction is a separate problem.

Investigate a bounded generic physical-headroom/library-allocation contract with mixed-grad plans, device/stream behavior and normal downstream workloads. Prefer trainer_rank-local work; no art.megatron change is proposed. Do not silently globalize the workaround based on one fixture.

Evidence: /home/brad/.local/share/schulman/retail49-memory-failure-20260909/MORNING-ALLOCATOR.md and replay-{observe,release}-comparison/. Local Caladan mitigation 431e424 and timing-only follow-up 177a9fce. Raw failures and cleanup receipts are preserved.

Start with the TrainerRank admission path and the micro_batch_loss.backward boundary, then read merged PR #900 and held PR #888 alongside the retained evidence paths named in the issue. Compare the measured cache-release behavior with the remaining backward, multi-device, throughput, and adoption gaps. Done requires a decided generic policy with the broader qualification still identified as necessary.