Every 403 on xorb fetch is treated as URL expiry; body discarded, 0 bytes written for minutes before aborting

## Summary

When the xorb CDN returns `403` for a reason that refreshing the presigned URL cannot fix, `hf_xet` still treats it as an expired URL, re-fetches the reconstruction, retries against a fresh URL, and gets the same `403`. It burns the whole retry budget this way while committing **zero bytes**, so every progress indicator sits at `0` and the transfer is indistinguishable from a hang. The `403` response body — which carries the server's actual diagnostic — is discarded and never logged.

This is a robustness/diagnosability report about the client, not a request to fix the edge. The specific `403` I hit may well be an infrastructure issue (see "Trigger" below), but the client behaviour turns a clear server error into a silent multi-minute stall, which is what made it undiagnosable.

## Client behaviour

`xet_client/src/cas_client/remote_client.rs:606` (v1.6.0):

```rust
let response = http_client.get(url).header(RANGE, range_header_value)
.with_extension(Api(api_tag)).send().await?;

if response.status() == reqwest::StatusCode::FORBIDDEN {
url_info.refresh_url().await
.map_err(|e| reqwest_middleware::Error::Middleware(e.into()))?;
}

Ok(response)
```

Two things here:

1. **Every `403` is assumed to be expiry.** There's no check that the URL was actually near expiry, and no discrimination between `403` causes. `process_ok_response` then classifies it `RetryableError` (`retry_on_403` is enabled for `s3::get_range` via `remote_client.rs:572`), so each attempt does a full reconstruction re-fetch plus a retry.

2. **The response body is dropped.** The `403` is surfaced only as `HTTP status client error (403 Forbidden)`. In my case the body was:

```
Auth failed: ParsingError: expected range header
```

I only found that by replaying the presigned URL by hand with `curl`. Logging the body (it's short and non-sensitive) would have made this self-diagnosing.

To be clear about what is *not* wrong: the retry is bounded — `exponential_retry_delays(base_delay, max_attempts, max_duration)` at `retry_wrapper.rs:272`, defaults 5 attempts / 3 s base / 360 s — and it does eventually log `No more retries; aborting`. I confirmed that empirically. The problem is the several minutes of apparent hang with no bytes and no actionable message beforehand, repeated per failing block.

## Observed log

Steady-state cycle, one `nvfp4_experts-*.safetensors` file (4.25 GB), `enable_multirange_fetching = false` (default):

```
INFO Retry on 403 (Forbidden) enabled): "s3::get_range" api call failed (request id 01M1EMKMN2FTTNA8WWAH79QJE5, retry 1): HTTP status client error (403 Forbidden)
INFO Received CAS response request_id=01M1EMKVAEF669JTTAVHKN2BCE status_code=403
INFO Refreshing expired retrieval URLs file_hash=eaa09f6b... byte_range=(0, 256000000) url_count=623
INFO Starting get_reconstruction API call call_id=16 bytes_range=Some(Range { start: 0, end: 256000000 }) api_version=v2
INFO Received CAS response request_id=01M1EMKVQ9P5BE69STYJFZBS17 status_code=206
INFO Retrieval URLs refreshed successfully
INFO Retry on 403 (Forbidden) enabled): "s3::get_range" api call failed (request id 01M1EMKVAEF669JTTAVHKN2BCE, retry 3): HTTP status client error (403 Forbidden)
...
```

Counts from one ~10 minute run of a single file:

| event | count |
|---|---|
| `s3::get_range` succeeded | 240 |
| `s3::get_range` 403 | 21 |
| `Refreshing expired retrieval URLs` | 21 |
| `Retry strategy` (i.e. `run_and_process` invocations) | 269 |
| `No more retries; aborting` | 1 |
| **bytes written to the output blob** | **0** |

`get_range` successes plateau at 240 while the 403s keep accumulating: the first 256 MB prefetch block contains failing xorbs, so it never completes and nothing is committed.

## Trigger (for context)

Xorb fetches route to `us.aws.cdn.hf.co/xorbs/default/?...&Policy=...&Signature=...&Key-Pair-Id=...`. Large signed ranges are rejected, small ones are fine:

```
# exact url_range from fetch_info, fresh reconstruction (Expires - now = 3599s)
67.10 MB -> 403 Auth failed: ParsingError: expected range header
67.09 MB -> 403
0.02 MB -> 206
0.01 MB -> 206
```

Also on `Qwen/Qwen3-8B` / `model-00001-of-00005.safetensors`, so it isn't repo-specific:

```
33.70 MB -> 403
25.08 MB -> 206
34.43 MB -> 403
4.73 MB -> 403
9.41 MB -> 206
2.81 MB -> 206
```

So it correlates with size but isn't a clean threshold — there's an intermittent component.

Things I checked and ruled out, in case they save someone time:

- **Not URL expiry.** A fresh URL with 3599 s of validity 403s immediately on its full signed range.
- **Not `Range` being stripped in transit.** `huggingface.co//resolve//` honours `Range` and returns `206` + `Content-Range` at sizes up to 128 MB from the same host; a header-echo origin sees `Range: bytes=0-1023` intact. The `expected range header` text appears to be a mis-tagged generic auth failure — a request with *no* `Range` header at all returns the identical message.
- **Not TLS / CA trust.** `Loaded 130 CA root certificates from the system`; control plane and small-range fetches over the same TLS work.
- **Not multi-range.** `enable_multirange_fetching = false`.
- **Not client disk/filesystem.** Local ext4, >1 TB free.

A failing request id, if the edge logs are inspectable: `01M1EN28FM2GTJ0969FTM6XPC0` (`x-hf-cdn-pop: aws-eu-west-3`).

This looks like the same family as #897 (GCP edge, `SignatureError: invalid key pair id`) and #800, and as huggingface/transformers#45797, which reports the identical log line and `us.aws.cdn.hf.co/xorbs/` 403s from a bare AWS `p5en.48xlarge` in `us-east-2` with no corporate proxy. All are closed.

## Why this mattered more than it should have

The repo I was fetching contains a 102.4 GB file. With Xet stalling, the documented fallback `HF_HUB_DISABLE_XET=1` doesn't apply either, because `huggingface_hub` refuses files above `MAX_HTTP_DOWNLOAD_SIZE` (50 GB) on the plain HTTP path and points back at `hf_xet`. So the repo had no working download path in the standard tooling. Plain parallel ranged `GET`s against the `resolve` endpoint worked fine at ~45–63 MB/s, which is what I ended up using.

## Suggestions

1. **Log the `403` body** (truncated). Single highest-value change here — it turns this from "silent hang" into "the server told you why".
2. **Don't treat every `403` as expiry.** Either check the URL's `Expires` before refreshing, or cap consecutive refreshes that produce no change in outcome and fail fast with the server's message.
3. **Surface repeated non-progress.** If a reconstruction block has committed 0 bytes across N refreshes, warn rather than continuing to look idle.
4. Optionally, **fall back to smaller sub-ranges** when a large range 403s — analogous to #842's single-range fallback. Small ranges succeeded consistently here, so this specific failure would have self-healed.

## Environment

```
hf_xet 1.6.0
huggingface_hub 1.29.0
python 3.10.12
OS Linux 6.8.0-138-generic (x86_64)
filesystem ext4, local, >1 TB free
repo Inferact/Qwen3.8-Flash-Next-NVFP4 (public), also Qwen/Qwen3-8B
auth unauthenticated (a token made no difference)
```

Corporate TLS interception is present on this network, but as noted above the control plane, small-range fetches, and `hf_xet`'s system-root loading all work over it, and #45797 reports the same failure without any proxy.