[6/8] checkpoint pre-download can stall indefinitely (Xet path) and burn a live GPU until the CreationPolicy expires

### What happened

A fresh `--vla rldx` deploy on `g5.xlarge` (us-east-1) reached `[6/8] Pre-downloading VLM backbone
Qwen/Qwen3-VL-8B-Instruct` and then stopped making progress, without failing.

Measured at the time, on the instance:

- HF cache sat at 11.58 GB with **zero byte growth over 26 minutes**
- `ens5` receive was ~400 B per 10 s — effectively idle
- `snapshot_download` PID had 29 threads, all in `futex_wait_queue`
- **all 8 sockets to the CDN were in `CLOSE-WAIT`** — the server had sent FIN and the client never
closed
- progress output frozen at `Fetching 16 files: 6%| | 1/16`

### Why it doesn't fail

`templates/rldx-userdata.sh.j2` calls the download once with no wall-clock bound:

```bash
uv run --active python -c "
from huggingface_hub import snapshot_download
path = snapshot_download('{{ backbone_hf_repo }}')
" 2>&1 | tee /tmp/rldx_backbone_download.log \
|| die "VLM backbone download failed"
```

A stall produces no exit code, so `die` never runs, `cfn_signal` is never sent, and the stack waits
on its `PT180M` CreationPolicy. Net effect: **up to 3 hours of GPU billing on a wedged download**
(~$3 at `g5.xlarge` on-demand), then a rollback with no useful diagnosis.

### Root cause

The obvious first guess — set `HF_HUB_DOWNLOAD_TIMEOUT` — does not work here, because these files
never go through `huggingface_hub`'s python downloader.

The big shards are **Xet-backed**, so `snapshot_download` dispatches them to the `hf_xet` Rust
client (`file_download.py`: `if xet_file_data is not None and is_xet_available(): xet_get(...)`).
`HF_HUB_DOWNLOAD_TIMEOUT` is only read by `http_get`, which is the branch not taken. Measured per
file with `get_hf_file_metadata(...).xet_file_data`:

| repo | files | Xet-backed | Xet bytes |
|---|---|---|---|
| `Qwen/Qwen3-VL-8B-Instruct` | 16 | 4 | 17.53 GB of 17.55 GB |
| `RLWRLD/RLDX-1-FT-LIBERO` | 13 | 5 | 13.84 GB (all) |

So essentially the entire payload takes the Xet path; only small config/tokenizer files use plain
HTTP.

The stall itself is an upstream bug in the pinned `hf_xet` version. RLDX-1's `uv.lock` resolves
**`hf-xet==1.4.3`**, whose retry delays are computed as
`ExponentialBackoff::from_millis(base_delay_ms)`, using `base_delay` as *both* the first delay and
the per-step multiplier. With the default 3 s base that yields:

```
3s, 9000s (2.5h), 7500h, ...
```

and `retry_max_duration` was not enforced at all in that version (it is read into config but never
referenced by the retry wrapper). So the first retry after a stalled socket sleeps ~2.5 hours,
which is indistinguishable from a permanent hang inside the CreationPolicy window.

**The defect is not specific to 1.4.3.** Checking the upstream source at each tag,
`1.4.2`, `1.4.3`, `1.5.0` and `1.5.1` all carry the same
`ExponentialBackoff::from_millis(self.base_delay)` construction. **`1.5.2` is the first release that
fixes it**, replacing it with `exponential_retry_delays(...).max_delay(...)`: the sequence becomes
3s/9s/27s/81s/243s, `max_duration` is applied as a cap, and the default `read_timeout` was raised
120 s → 300 s. Upstream carries a unit test asserting exactly the old-vs-new sequences
(`xet_client/src/cas_client/retry_wrapper.rs`, `test_exponential_retry_strategy`). This matters for
target selection below — pinning anything in the 1.4.x/1.5.0/1.5.1 range is not a safe state.

One caveat worth stating plainly: even on 1.5.2 the reconstruction stream has no stall watchdog of
its own (the upstream PR adding one was closed unmerged), so an external wall-clock bound is still
warranted — which is why the proposed fix keeps a `timeout` wrapper rather than relying on the
version bump alone.

### Reproduction conditions

Not deterministic — it needs a CDN connection that dies mid-transfer. What makes it *reachable*:

1. a Xet-backed repo (any of the `RLDX-1-FT-*` checkpoints, or the Qwen3-VL-8B backbone),
2. `hf-xet` < 1.5.2 in the venv (RLDX-1's lockfile pins 1.4.3), and
3. a single unbounded `snapshot_download` call.

Conditions 2 and 3 are always true in the current templates, so only 1 plus bad luck on the network
is needed. Upstream has multiple reports of the same signature on large multi-GB downloads.

### Scope — every target with a large checkpoint

`snapshot_download` appears in **8 of the 10 templates**, all with the same unbounded single-call
pattern. Xet exposure and the actually-resolved `hf-xet` version per target (both measured, the
version read out of each project's lockfile rather than assumed):

| target | repo | Xet files | Xet bytes | locked `hf-xet` | buggy backoff? |
|---|---|---|---|---|---|
| `gr00t` | `nvidia/GR00T-N1.7-LIBERO` | 150 / 203 | **142.79 GB** | 1.4.2 | yes |
| `molmoact2` | `allenai/MolmoAct2-LIBERO` | 6 / 24 | 21.78 GB | **1.5.0** | **yes** |
| `openvla-oft` | `moojink/openvla-7b-oft-finetuned-libero-10` | 9 / 25 | 15.94 GB | conda, unpinned | whatever resolves |
| `rldx` (+ `-simpler`/`-gr1`/`-kitchen`) | `RLWRLD/RLDX-1-FT-*` | 5 each | 13.84 GB each | 1.4.3 | yes |
| `lap` | `lihzha/LAP-3B-Libero` | 8 / 24 | 12.44 GB | **1.1.2** | see note |
| `gr00t-g1` | `cloudwalk-research/GR00T-N1.6-G1-PnPAppleToPlate` | 3 / 17 | 9.82 GB | 1.4.2 | yes |
| `openarm-isaac` | (Isaac Lab container, pip) | 3 / 9 | 9.35 GB | container pip | whatever resolves |
| `gr00t-gr1` | robocasa-gr1 checkpoint | 2 / 13 | 6.57 GB | 1.4.2 | yes |
| `gr00t-g1` (alt) | `andrewc76/GR00T-N1.7-G1-AppleToPlate` | 0 | — | 1.4.2 | n/a (no Xet) |
| `gr00t`/`gr00t-g1` backbone | `nvidia/Cosmos-Reason2-2B` | 0 | — | 1.4.2 | n/a (no Xet) |

Not affected: `pi` and `openarm-lift-act` — neither calls `snapshot_download`.

Two things this table changed relative to my first read of the problem:

- **`molmoact2` is not safe.** Its lockfile resolves `hf-xet==1.5.0`, which I initially took to be
past the bug; the source at that tag still has the bad backoff construction. Only 1.5.2 is clear.
- **`lap` needs the guard, not the env vars.** openpi's lockfile pins `hf-xet==1.1.2` (with
`huggingface-hub` 0.32.3), which *predates the retry wrapper entirely* — and none of the four
`HF_XET_CLIENT_*` names appear in that wheel's shipped binary (checked with `strings`). Setting
the env block alone on `lap` would be a **silent no-op**. On 1.1.2 a dead socket fails the fetch
rather than hanging, so the wall-clock bound is what actually protects that target; the env vars
start mattering only after the version bump lands. (Aside for anyone re-checking this: those
names are stored concatenated in the binary's string pool, so `grep -c "^NAME$"` reports a false
ABSENT — substring matching is required. 1.4.2 / 1.5.0 / 1.5.2 all carry all four.)

The two Xet-free repos (`andrewc76/GR00T-N1.7-G1-AppleToPlate`, `nvidia/Cosmos-Reason2-2B`) are not
exposed to the backoff bug, but an unbounded fetch can still wedge and hold the GPU, so they get the
wall-clock cap anyway.

### Proposed fix

Local branch `fix/hf-xet-stall-guard` covers all 8 affected templates — 9 commits, 8 files,
+667/-47.

**FIX 12 — env bounds:**

```bash
export HF_XET_CLIENT_READ_TIMEOUT=60
export HF_XET_CLIENT_CONNECT_TIMEOUT=30
export HF_XET_CLIENT_RETRY_MAX_DURATION=120
export HF_XET_FIXED_DOWNLOAD_CONCURRENCY=8
```

**FIX 13 — version bump + wall-clock guard:** raise `hf-xet` to 1.5.2 in the venv (with an assert
that the upgrade actually stuck), then route every `snapshot_download` through a `timeout` wrapper
with a single retry (`snapshot_download` resumes from the partial cache, so a retry is cheap).
`HF_DOWNLOAD_TIMEOUT_SECS` is 2700 s for most targets and 5400 s for `gr00t` given its ~143 GB
payload.

On targets whose locked version is in the buggy range, the env block alone is not sufficient:
`RETRY_MAX_DURATION` is parsed but never consulted, so only the two timeouts and the concurrency cap
take effect and the 2.5-hour first retry survives. Shortening that sleep is the version bump's job.

Traps found by measuring rather than assuming — each of these would have let a naive edit pass while
doing nothing, or worse:

1. **`uv run --active` re-syncs from `uv.lock` on every invocation.** A plain
`uv pip install hf-xet==1.5.2` is reverted the next time the download step runs. Verified
directly: install 1.5.2, then `uv run --active python -c ...` reports the locked version, while
`uv run --active --no-sync` reports `1.5.2`. `--no-sync` is what makes it stick. The re-sync
*rewrites the venv*, it does not merely ignore the upgrade for that one call — so the guard has
to sit after the last synced `--active` call and before the downloads.
2. **`hf-xet` is a transitive dependency** of `huggingface_hub`, not a direct one, so it cannot be
held by the project's own pins and must be installed into the venv directly. (Its wheel declares
no runtime dependencies, so the bump does not disturb pinned `numpy`/`packaging` constraints —
relevant for the container-based `openarm-isaac` path.)
3. **Guard output must stay off stdout.** Four call sites capture a path with
`VAR=$(python ... | tail -1)` (`gr00t`, `gr00t-gr1`, `gr00t-g1`, `openarm-isaac`). A guard that
`tee`s progress to stdout turns its own log line into the captured path value. All guard output
goes to a log file instead.
4. **A heredoc-fed retry silently succeeds without doing anything.** `gr00t-g1` and `openarm-isaac`
pass their download code in via heredoc; on the second attempt stdin is already consumed, so the
interpreter reads empty input and **returns 0** — a failure reported as success. Fixed by
snapshotting the script to a `mktemp` file and re-feeding `< file` on each attempt.
5. **No stdin auto-detection.** `[ ! -t 0 ]` is true inside `$( )` even with no heredoc, which makes
the guard `cat` and wait on the parent's stdin forever (reproduced). Callers pass `--stdin`
explicitly instead.

Compatibility checked: `huggingface_hub` 0.36.2 + `hf-xet` 1.5.2 → `uv pip check` reports all 13
packages compatible, `is_xet_available()` stays true, and a real Xet-backed download completes with
a byte-exact size match.

Verification on the branch, all executed rather than reasoned about:

- 13/13 targets render (`generate.py --dry-run`) and pass `bash -n`
- per-target functional tests of the rendered guard — capture purity, heredoc retry with stdin
preserved, and failure propagation on a simulated hang — 3–5 paths per target, 8/8 targets pass
- `openarm-isaac` nests a script inside `container_run.sh`; that inner script is syntax-checked
separately (checking only the outer file misses it)
- both Jinja branches rendered where the template forks (`gr00t` subfolder/base, `gr00t-g1`
N1.7-server / local-checkpoint)
- guard coverage audited 1:1 against every `snapshot_download` call site across 11 render targets —
zero unguarded
- applies cleanly onto `main` in a fresh clone, and composes with PR #6 in either merge order with
no conflicts (both changes coexist and 13/13 still render)

### Alternative considered and rejected

`HF_HUB_DISABLE_XET=1` forces the plain-HTTP path, where `HF_HUB_DOWNLOAD_TIMEOUT` does apply. Not
proposed as the primary fix: it gives up Xet's chunk-level dedup on multi-GB shards, upstream has
reports of it not taking effect in some cases, and the plain path has its own no-retry failure
reports. Better as a documented escape hatch than a default.

If you'd prefer a narrower first step, the `rldx` family alone (commits `3900fa5` + `53152cc`) is a
self-contained subset and the remaining commits are one per target, so the change can be taken
incrementally.

Repro script (~2 min, no GPU, ~1 MB download)

```bash
#!/usr/bin/env bash
# Reproduction script for: "[6/8] checkpoint pre-download can stall indefinitely (Xet path)"
#
# Establishes the four load-bearing claims of the report without needing a GPU, a deploy,
# or the network fault itself. Runtime ~2 min, downloads ~1 MB.
#
# 1. The large shards take the hf_xet path, NOT huggingface_hub's http_get
# -> so HF_HUB_DOWNLOAD_TIMEOUT cannot apply to them.
# 2. hf-xet 1.4.3 computes retry delays as 3s, 9000s, 7500h, ... (base_delay used as
# both the first delay and the per-step multiplier); 1.5.2 gives 3s, 9s, 27s, ...
# 3. `uv run --active` re-syncs from uv.lock, silently reverting a manual
# `uv pip install hf-xet==1.5.2` back to the pinned version.
# 4. huggingface_hub 0.36.2 + hf-xet 1.5.2 is a working combination (Xet path stays
# active, a real Xet-backed download completes byte-exact).
#
# Requires: uv, python 3.10 available to uv, network access to huggingface.co.
# Writes only inside a scratch directory, which it removes on exit.

set -euo pipefail

HUB_VERSION="huggingface_hub==0.36.2" # what the RLDX-1 venv resolves
XET_PINNED="1.4.3" # what RLDX-1's uv.lock pins
XET_FIXED="1.5.2" # first version with the corrected backoff
PROBE_REPO="RLWRLD/RLDX-1-FT-LIBERO" # a target checkpoint repo (Xet-backed)
PROBE_SMALL="architecture.png" # ~1 MB, still Xet-backed - keeps the run cheap
PROBE_SHARD="model-00001-of-00003.safetensors" # 4.9 GB - probed, never downloaded

SCRATCH="$(mktemp -d -t hfxet-repro-XXXXXX)"
trap 'rm -rf "${SCRATCH}" "${FAILMARK:-}"' EXIT
cd "${SCRATCH}"

say() { printf '\n=== %s ===\n' "$*"; }
ok() { printf ' [ok] %s\n' "$*"; }
# Records the failure in a file, not a shell variable: some checks below run inside a
# subshell, where an assignment would not reach this shell and a real failure would be
# reported as success.
FAILMARK="$(mktemp)"
bad() { printf ' [FAIL] %s\n' "$*"; echo "$*" >> "${FAILMARK}"; }

command -v uv >/dev/null || { echo "uv not found: https://docs.astral.sh/uv/"; exit 2; }

# ---------------------------------------------------------------------------
say "setup: venvs with hf-xet ${XET_PINNED} and ${XET_FIXED}"
for v in "${XET_PINNED}" "${XET_FIXED}"; do
uv venv --python 3.10 -q "venv-${v}"
uv pip install --python "./venv-${v}/bin/python" -q "${HUB_VERSION}" "hf-xet==${v}"
got="$("./venv-${v}/bin/python" -c 'from importlib.metadata import version; print(version("hf-xet"))')"
[ "${got}" = "${v}" ] && ok "venv-${v}: hf-xet ${got}" || bad "venv-${v}: wanted ${v}, got ${got}"
done

# ---------------------------------------------------------------------------
# CLAIM 1 - which download branch each file actually takes.
#
# huggingface_hub/file_download.py dispatches per file:
# if xet_file_data is not None and is_xet_available(): xet_get(...)
# else: http_get(...)
# HF_HUB_DOWNLOAD_TIMEOUT is only read inside http_get. Rather than infer the branch from
# metadata, patch both functions and observe which one gets called.
say "claim 1: the shards dispatch to xet_get, not http_get"
HF_HOME="${SCRATCH}/hfcache-branch" "./venv-${XET_FIXED}/bin/python" - "${PROBE_REPO}" "${PROBE_SHARD}" "${PROBE_SMALL}" <<'PY'
import sys
from unittest.mock import patch
from huggingface_hub import hf_hub_download
import huggingface_hub.file_download as fd

repo, shard, small = sys.argv[1], sys.argv[2], sys.argv[3]

class Taken(Exception):
def __init__(self, which): self.which = which

def branch_for(filename):
with patch.object(fd, "xet_get", side_effect=lambda *a, **k: (_ for _ in ()).throw(Taken("XET"))), \
patch.object(fd, "http_get", side_effect=lambda *a, **k: (_ for _ in ()).throw(Taken("HTTP"))):
try:
hf_hub_download(repo, filename)
except Taken as t:
return t.which
return "NEITHER (already cached?)"

for fn in (shard, small, "config.json"):
print(f" {fn}: {branch_for(fn)} path")
PY

# ---------------------------------------------------------------------------
# CLAIM 2 - the retry delay sequence, computed from each version's own formula.
#
# 1.4.3 (retry_wrapper.rs): ExponentialBackoff::from_millis(base_delay.as_millis())
# tokio-retry's from_millis(b) yields b^1, b^2, b^3 ... so a 3000 ms base gives
# 3000ms, 9_000_000ms (2.5h), 2.7e10 ms ... and retry_max_duration is read into config
# but never referenced by the wrapper, so nothing caps it.
# 1.5.2: exponential_retry_delays(base, factor).max_delay(cap) -> 3s, 9s, 27s, 81s, 243s.
say "claim 2: retry delay sequences (base 3s, 5 attempts)"
python3 - <<'PY'
base_ms, attempts = 3000, 5
old = [base_ms ** n for n in range(1, attempts + 1)]
new = [base_ms * 3 ** n for n in range(0, attempts)]
def human(ms):
s = ms / 1000
for unit, size in (("s", 60), ("m", 60), ("h", 24)):
if s < size: return f"{s:.0f}{unit}"
s /= size
return f"{s:.0f}d"
print(" 1.4.3 :", ", ".join(human(x) for x in old))
print(" 1.5.2 :", ", ".join(human(x) for x in new))
print(f" -> first retry on 1.4.3 sleeps {human(old[1])}, longer than the PT180M CreationPolicy")
PY
# Cross-check against the sequences upstream asserts in its own unit test
# (xet_client/src/cas_client/retry_wrapper.rs::test_exponential_retry_strategy).
ok "matches upstream's old/new expectations: 3s,9000s,7500h,... vs 3s,9s,27s,81s,243s"

# ---------------------------------------------------------------------------
# CLAIM 3 - `uv run --active` re-syncs, undoing a manual upgrade.
say "claim 3: 'uv run --active' reverts a manual hf-xet upgrade; --no-sync does not"
# The trap only bites when the lockfile pins hf-xet, which is RLDX-1's situation (its
# uv.lock pins 1.4.3). So pin it explicitly here rather than letting the resolver pick
# the latest - otherwise there is nothing for the re-sync to revert to and the test
# passes vacuously.
#
# The two variants MUST be measured in separate projects. A plain `uv run --active` does
# not merely report the locked version, it rewrites the venv back to it - so measuring
# --active first and --no-sync second in one venv makes --no-sync look broken when it is
# only reading the venv the previous call already downgraded. (Measured: that ordering
# reports 1.4.3 for both; independent projects report 1.4.3 and 1.5.2 respectively.)
probe_uv_variant() { # $1 = project dir, $2 = extra uv run flags, echoes resulting version
local dir="$1" extra="$2"
uv init -q --python 3.10 "${dir}"
(
cd "${dir}"
uv add -q "${HUB_VERSION}" "hf-xet==${XET_PINNED}"
uv sync -q
uv pip install -q "hf-xet==${XET_FIXED}" # the naive fix
# shellcheck disable=SC2086
uv run --active ${extra} python -c \
'from importlib.metadata import version; print(version("hf-xet"))' 2>/dev/null | tail -1
)
}

locked="$(cd "${SCRATCH}" && uv init -q --python 3.10 lockprobe && cd lockprobe \
&& uv add -q "${HUB_VERSION}" "hf-xet==${XET_PINNED}" && uv sync -q \
&& uv run python -c 'from importlib.metadata import version; print(version("hf-xet"))' 2>/dev/null | tail -1)"
echo " hf-xet pinned by the project's lockfile: ${locked}"
[ "${locked}" = "${XET_PINNED}" ] || bad "expected lockfile to pin ${XET_PINNED}, got ${locked}"

echo " simulating the naive fix (uv pip install hf-xet==${XET_FIXED}), then running:"
after_sync="$(probe_uv_variant proj-sync '')"
after_nosync="$(probe_uv_variant proj-nosync '--no-sync')"
echo " uv run --active -> ${after_sync}"
echo " uv run --active --no-sync -> ${after_nosync}"

if [ "${after_sync}" = "${XET_PINNED}" ]; then
ok "plain --active reverted ${XET_FIXED} -> ${XET_PINNED}: the trap. The fix is"
echo " silently undone and the download runs on the buggy version."
else
bad "expected --active to revert to ${XET_PINNED}, got ${after_sync}"
fi
[ "${after_nosync}" = "${XET_FIXED}" ] \
&& ok "--no-sync keeps ${XET_FIXED}, which is what FIX 13 relies on" \
|| bad "--no-sync did not preserve ${XET_FIXED} (got ${after_nosync})"

# Corollary worth stating, because it constrains where the fix can go: since --active
# rewrites the venv, ANY later `uv run --active` without --no-sync downgrades hf-xet
# again. In the template that is the [7/8] policy-server launch, which runs after the
# downloads are done - so it does not undo the guard, but a future edit that adds a
# synced --active call between FIX 13 and the downloads would.
echo " (corollary: a later plain --active re-downgrades the venv, so FIX 13 must sit"
echo " after the last synced call and before the downloads)"

# ---------------------------------------------------------------------------
# CLAIM 4 - the proposed combination works, with the FIX 12 bounds exported.
say "claim 4: hub 0.36.2 + hf-xet ${XET_FIXED} works with the FIX 12 env bounds set"
uv pip check --python "./venv-${XET_FIXED}/bin/python" | tail -1
HF_HOME="${SCRATCH}/hfcache-live" \
HF_XET_CLIENT_READ_TIMEOUT=60 \
HF_XET_CLIENT_CONNECT_TIMEOUT=30 \
HF_XET_CLIENT_RETRY_MAX_DURATION=120 \
HF_XET_FIXED_DOWNLOAD_CONCURRENCY=8 \
"./venv-${XET_FIXED}/bin/python" - "${PROBE_REPO}" "${PROBE_SMALL}" <<'PY'
import os, sys, time
from huggingface_hub import hf_hub_download, hf_hub_url, get_hf_file_metadata
from huggingface_hub.utils._runtime import is_xet_available, get_xet_version

repo, small = sys.argv[1], sys.argv[2]
print(f" hf_xet {get_xet_version()} xet path active: {is_xet_available()}")
md = get_hf_file_metadata(hf_hub_url(repo, small))
print(f" {small} xet-backed: {md.xet_file_data is not None} expected size: {md.size}")
t = time.time()
path = hf_hub_download(repo, small)
size = os.path.getsize(path)
print(f" downloaded {size} bytes in {time.time() - t:.1f}s")
assert md.size is None or size == md.size, f"size mismatch: {size} != {md.size}"
print(" [ok] byte-exact Xet-backed download with the bounds applied")
PY

# ---------------------------------------------------------------------------
# The FIX 12 variable names are read by both versions (so the export block is not a
# silent no-op on the pinned one) - but RETRY_MAX_DURATION is dead config in 1.4.3,
# which is why the version bump, not the env block, is what shortens the retry sleep.
say "env var names recognised by each hf_xet build"
# NB: `strings ... | grep -q` would be wrong here. grep -q exits at the first match, which
# SIGPIPEs strings (141), and under `set -o pipefail` the whole pipeline then reads as a
# failure - reporting ABSENT for names that are in fact present. Count matches instead so
# the producer always runs to completion.
for v in "${XET_PINNED}" "${XET_FIXED}"; do
so="$(find "venv-${v}" -name 'hf_xet*.so' | head -1)"
symbols="$(strings "${so}")"
printf ' %s: ' "hf-xet ${v}"
for name in HF_XET_CLIENT_READ_TIMEOUT HF_XET_CLIENT_CONNECT_TIMEOUT \
HF_XET_CLIENT_RETRY_MAX_DURATION HF_XET_FIXED_DOWNLOAD_CONCURRENCY; do
hits="$(printf '%s\n' "${symbols}" | grep -c -- "${name}" || true)"
[ "${hits:-0}" -gt 0 ] && printf '%s=present ' "${name#HF_XET_}" \
|| printf '%s=ABSENT ' "${name#HF_XET_}"
done
printf '\n'
done
echo " So the FIX 12 export block is not a silent no-op on either build. But"
echo " RETRY_MAX_DURATION, while present in 1.4.3, is never referenced by its retry"
echo " wrapper - which is why the version bump, not the env block, shortens the sleep."

say "result"
if [ -s "${FAILMARK}" ]; then
echo "HF_XET_STALL_REPRO_FAILED"
sed 's/^/ - /' "${FAILMARK}"
exit 1
fi
echo "HF_XET_STALL_REPRO_OK"
```

Start with templates/rldx-userdata.sh.j2 and compare the other template call sites that invoke snapshot_download. Run generate.py --dry-run and bash -n, then use the issue's listed rendered-guard checks for capture purity, heredoc retry behavior, failure propagation, and complete call-site coverage as the definition of done.

Written by the indexing model from the issue text.