voxora-hf verify_sha256_sidecars reads entire target file into memory
- Vorherrschende Sprache
- Rust
- Sterne
- 0
- Forks
- 1
- Ø Merge
- 11 Min.
- Gemergte PRs (30 T.)
- 77
Beschreibung
## What
`voxora-hf/src/source.rs:693-697` (`verify_sha256_sidecars`)
reads the full target file into RAM before hashing:
```rust
let bytes = std::fs::read(&target_path).map_err(|e| HfError::Io { ... })?;
let mut hasher = Sha256::new();
hasher.update(&bytes);
```
A sharded model whose `model-00001-of-00002.safetensors.sha256`
sidecar exists will read the full shard into RAM before hashing.
On a 10 GB shard that is a 10 GB allocation. The verify runs
serially inside `CacheResolver::run` (right after download) so
peak RSS at end of a successful resolve is approximately
"size of the largest shard with a sidecar".
## Why it matters
The function has zero direct test coverage today. The
`mark_complete` path is exercised through
`voxora-hf/tests/wiremock_sharded.rs`, but a sharded model with
a `.sha256` sidecar (Qwen3-ASR, Llama-3-ASR variants) trips this
allocation. On a 32 GB laptop resolving a 24 GB Qwen3-ASR split
across 4 shards, the verify step alone pushes the process over
16 GB resident.
## Recipe
Swap `std::fs::read` for a buffered `tokio::fs::File` +
`tokio::io::copy` into a streaming `Sha256` hasher
(`sha2::Digest::new()` accepts streaming updates). Concretely:
```rust
let mut file = tokio::fs::File::open(&target_path).await?;
let mut hasher = Sha256::new();
let mut buf = vec![0u8; 64 * 1024];
loop {
let n = file.read(&mut buf).await?;
if n == 0 { break; }
hasher.update(&buf[..n]);
}
```
Add a direct unit test for `verify_sha256_sidecars` that writes
a multi-MB target file, a sidecar with the correct hash, and
asserts the function returns `Ok`. A second test that tampers
with the sidecar to a wrong hash asserts the function returns
`Err` (today the function silently succeeds if no sidecar
exists; that contract should also be pinned by a test).
## Acceptance
- `verify_sha256_sidecars` streams the target file.
- Peak RSS during a Qwen3-ASR 24 GB resolve stays below
500 MB (regardless of shard size).
- New unit tests cover the happy path and the sidecar-tampered
path.
Beitragsleitfaden
Rechercherichtung
Start at voxora-hf/src/source.rs:693-697 and inspect verify_sha256_sidecars, then review the related coverage in voxora-hf/tests/wiremock_sharded.rs. Replace the whole-file hashing path with streaming reads, add direct tests for a correct and tampered sidecar, and confirm the existing sharded-model flow still passes.
Vom Indexierungsmodell aus dem Issue-Text verfasst.
Bewertung
- Tech-Stack
- rust
- Bereich
- performance
- Issue-Typ
- Bug
- Schwierigkeit
- 3/5
- Geschätzter Aufwand
- 1-2 Tage
- Aktivitätsstatus
- Aktiv
- Klarheit
- Klar beschrieben
- Anfängerfreundlichkeit
- 76/100