airvzxf / airvzxf/voxora

voxora-hf verify_sha256_sidecars reads entire target file into memory

Geschlossen
#111 1 Kommentar 0 Reaktionen 0 zugewiesene Personen Auf GitHub ansehen
bug enhancement performance
Vorherrschende Sprache
Rust
Sterne
0
Forks
1
Ø Merge
11 Min.
Gemergte PRs (30 T.)
77

Beschreibung

## What

`voxora-hf/src/source.rs:693-697` (`verify_sha256_sidecars`)
reads the full target file into RAM before hashing:

```rust
let bytes = std::fs::read(&target_path).map_err(|e| HfError::Io { ... })?;
let mut hasher = Sha256::new();
hasher.update(&bytes);
```

A sharded model whose `model-00001-of-00002.safetensors.sha256`
sidecar exists will read the full shard into RAM before hashing.
On a 10 GB shard that is a 10 GB allocation. The verify runs
serially inside `CacheResolver::run` (right after download) so
peak RSS at end of a successful resolve is approximately
"size of the largest shard with a sidecar".

## Why it matters

The function has zero direct test coverage today. The
`mark_complete` path is exercised through
`voxora-hf/tests/wiremock_sharded.rs`, but a sharded model with
a `.sha256` sidecar (Qwen3-ASR, Llama-3-ASR variants) trips this
allocation. On a 32 GB laptop resolving a 24 GB Qwen3-ASR split
across 4 shards, the verify step alone pushes the process over
16 GB resident.

## Recipe

Swap `std::fs::read` for a buffered `tokio::fs::File` +
`tokio::io::copy` into a streaming `Sha256` hasher
(`sha2::Digest::new()` accepts streaming updates). Concretely:

```rust
let mut file = tokio::fs::File::open(&target_path).await?;
let mut hasher = Sha256::new();
let mut buf = vec![0u8; 64 * 1024];
loop {
let n = file.read(&mut buf).await?;
if n == 0 { break; }
hasher.update(&buf[..n]);
}
```

Add a direct unit test for `verify_sha256_sidecars` that writes
a multi-MB target file, a sidecar with the correct hash, and
asserts the function returns `Ok`. A second test that tampers
with the sidecar to a wrong hash asserts the function returns
`Err` (today the function silently succeeds if no sidecar
exists; that contract should also be pinned by a test).

## Acceptance

- `verify_sha256_sidecars` streams the target file.
- Peak RSS during a Qwen3-ASR 24 GB resolve stays below
500 MB (regardless of shard size).
- New unit tests cover the happy path and the sidecar-tampered
path.

Beitragsleitfaden

Beitragsleitfaden öffnen

Rechercherichtung

Start at voxora-hf/src/source.rs:693-697 and inspect verify_sha256_sidecars, then review the related coverage in voxora-hf/tests/wiremock_sharded.rs. Replace the whole-file hashing path with streaming reads, add direct tests for a correct and tampered sidecar, and confirm the existing sharded-model flow still passes.

Vom Indexierungsmodell aus dem Issue-Text verfasst.

Bewertung

Tech-Stack
rust
Bereich
performance
Issue-Typ
Bug
Schwierigkeit
3/5
Geschätzter Aufwand
1-2 Tage
Aktivitätsstatus
Aktiv
Klarheit
Klar beschrieben
Anfängerfreundlichkeit
76/100

Neue Issues direkt in Ihr Postfach

Eine kurze Übersicht über anfängerfreundliche GitHub-Issues.