[performance][v0.8.15] Reusable SSD loads hash full model weights twice (25.1 TB read in 49h)

## Component

Provider (Swift), reusable SSD prefix-cache model loading / `WeightHasher`

## What happened?

On v0.8.15, every reusable-SSD cold model load performs two fresh SHA-256 passes over all model weight files: one before container loading and one after. On a provider advertising five 12.1–21.3 GB models, normal coordinator-driven slot churn produced **1,401 complete full-weight scans in 49.06 hours**: **5.72 hours of scan wall-time and an estimated 25.07 TB read**.

The two-pass TOCTOU invariant is intentional and security-sensitive. This issue is not proposing that the post-load check simply be deleted. The problem is that the invariant's current cost is paid again on every cold reload even though all five models retained one stable digest throughout the observation window.

## Environment

- Darkbloom: `0.8.15`
- macOS: `26.5.2` (`25F84`)
- Hardware: Apple M3 Max (`Mac15,9`), 128 GB unified memory
- Launch configuration: five explicit `--model` arguments, default `max_model_slots = 3`
- Prefix cache: reusable SSD path enabled
- Observation window: `2026-08-29T04:13:32Z` through `2026-08-31T05:17:15Z` (49.06 hours)
- Samples: 100,583 system samples at roughly 2-second cadence, 11,637 successful inference attempts in the overlapping window

## Evidence

The provider log's `Computing weight hash...` and `Weight hash for...` messages were paired by model and timestamp. Consecutive completed scans for the same model with a gap of at most five seconds were classified as the pre/post pair around one load.

- Completed scans: **1,401**
- Paired load cycles: **692**
- Scans belonging to pairs: **1,384 / 1,401 (98.8%)**
- Median paired cycle: **31 seconds**; p95: **39 seconds**
- Digest mismatches within pairs: **0**
- Distinct completed digests per model over the entire window: **1 each**
- Scan wall-time: **5.72 hours**, or 11.7% of the wall-clock window
- Estimated bytes read: **25.07 TB**, computed as scan count × catalog `total_size_bytes`

| Model | Full scans | Median / p95 scan | Scan hours | Estimated TB read | Distinct digests |
|---|---:|---:|---:|---:|---:|
| `gemma-4-26b-qat-4bit` | 187 | 12s / 15s | 0.65 | 2.92 | 1 |
| `gpt-oss-20b` | 221 | 10s / 11s | 0.61 | 2.68 | 1 |
| `qwen3-vl-30b-a3b-instruct` | 493 | 15s / 17s | 2.06 | 9.01 | 1 |
| `qwen3.5-35b-a3b` | 457 | 17s / 20s | 2.19 | 9.55 | 1 |
| `qwen3.6-35b-a3b-vl-mtp-mxfp8` | 43 | 17s / 21s | 0.21 | 0.92 | 1 |

The load-pair counts also match independently sampled warm-set additions almost exactly by model (692 pairs versus 694 additions, including initial left-edge residents), which is evidence that these are load-bracketing scans rather than unrelated startup checks.

### CPU hypothesis test

I aggregated system telemetry into 590 five-minute bins and fit:

```text
system_cpu_usage ~ hash_duty + inference_active_duty + model_churn_events + elapsed_time
```

with Newey–West HAC standard errors (12 lags). Full hash duty was associated with **+0.092 system CPU fraction** (about 9.2 percentage points), `t = 2.676`, raw `p = 0.007`, Benjamini–Hochberg `q = 0.010` across 41 exploratory tests. This is observational, but the source and log timing provide a direct mechanism.

## Source-level explanation

In the v0.8.15 source, a reusable SSD load explicitly sets `requireFreshCryptographicHash` for the pre-load observation and performs another fresh observation after loading:

- [`ProviderLoop+ModelLoading.swift` lines 362–402](https://github.com/Layr-Labs/d-inference/blob/0e63aed56a2e0956650dbc91ae9c3edfe6e04b89/provider-swift/Sources/ProviderCore/ProviderLoop%2BModelLoading.swift#L362-L402)

`WeightHasher` collects every integrity file and streams each complete file through SHA-256:

- [`WeightHasher.swift` lines 42–57](https://github.com/Layr-Labs/d-inference/blob/0e63aed56a2e0956650dbc91ae9c3edfe6e04b89/provider-swift/Sources/ProviderCoreFoundation/WeightHasher.swift#L42-L57)
- [`WeightHasher.swift` lines 186–205](https://github.com/Layr-Labs/d-inference/blob/0e63aed56a2e0956650dbc91ae9c3edfe6e04b89/provider-swift/Sources/ProviderCoreFoundation/WeightHasher.swift#L186-L205)

There is already a cheap path/size/mtime fingerprint whose documentation says an unchanged fingerprint can skip a full re-read for honest-provider drift detection, but the reusable SSD branch deliberately bypasses it:

- [`WeightHasher.swift` lines 60–90](https://github.com/Layr-Labs/d-inference/blob/0e63aed56a2e0956650dbc91ae9c3edfe6e04b89/provider-swift/Sources/ProviderCoreFoundation/WeightHasher.swift#L60-L90)

## Steps to reproduce

1. Enable reusable SSD prefix caching.
2. Advertise more models than the provider's effective `max_model_slots`, so normal traffic causes cold reloads.
3. Run v0.8.15 under coordinator traffic for several hours.
4. Extract the hash lifecycle messages:

```bash
rg 'Computing weight hash|Weight hash for' ~/.darkbloom/provider.log
```

5. Pair each `Computing` record with its same-model completion, then pair consecutive completed same-model scans separated by ≤5 seconds.
6. Join each model with `total_size_bytes` from `darkbloom models --json`; sum `scan_count * total_size_bytes`.
7. Verify that each cold load normally has two complete scans and that the digests remain unchanged.

## Expected behavior

Reusable SSD cache identity and mutation detection should remain fail-closed, but repeated loads of the same immutable catalog artifact should not require tens of terabytes of full-file reads over two days.

Possible safe directions to evaluate:

1. Bind a verified immutable artifact snapshot to stable file descriptors/inodes and load from that same snapshot, so one verification can cover the bytes actually loaded.
2. Leverage the signed/immutable catalog artifact identity plus filesystem immutability or a mutation watcher, falling back to fresh full hashes whenever identity or metadata changes.
3. Preserve the two-observation check only where the artifact can actually mutate during the load window, rather than paying two full passes on every eviction/reload.
4. Add counters for full hash scans, bytes read, duration, and reason (`startup`, `pre_load`, `post_load`, `fingerprint_drift`) so this cost is visible before it becomes SSD churn.

## Privacy / confirmation

The analysis used aggregate timings, model IDs, resource samples, and catalog sizes only. It excludes prompts, request bodies, account/provider identifiers, wallet data, hostnames, IP addresses, usernames, keys, and local paths.

- [x] I searched open and closed issues for weight hashing, SSD cache hashing, and cold-load hashing; I did not find a dedicated report for this repeated two-pass cost.

Read ProviderLoop+ModelLoading.swift lines 362–402 and WeightHasher.swift lines 42–90 and 186–205 first. Compare the reusable SSD branch with the existing fingerprint path, then evaluate how a safe change preserves the fail-closed two-observation invariant while avoiding repeated full reads for unchanged artifacts; done means mutation detection remains reliable and scan cost is observable.

Written by the indexing model from the issue text.