airvzxf / airvzxf/voxora

voxora-hf verify_sha256_sidecars reads entire target file into memory

Open
#111 1 comment 0 reactions 0 assignees View on GitHub
bug enhancement performance
Dominant language
Rust
Stars
0
Forks
1
Avg merge
11m
Merged PRs (30d)
47

Description

## What

`voxora-hf/src/source.rs:693-697` (`verify_sha256_sidecars`)
reads the full target file into RAM before hashing:

```rust
let bytes = std::fs::read(&target_path).map_err(|e| HfError::Io { ... })?;
let mut hasher = Sha256::new();
hasher.update(&bytes);
```

A sharded model whose `model-00001-of-00002.safetensors.sha256`
sidecar exists will read the full shard into RAM before hashing.
On a 10 GB shard that is a 10 GB allocation. The verify runs
serially inside `CacheResolver::run` (right after download) so
peak RSS at end of a successful resolve is approximately
"size of the largest shard with a sidecar".

## Why it matters

The function has zero direct test coverage today. The
`mark_complete` path is exercised through
`voxora-hf/tests/wiremock_sharded.rs`, but a sharded model with
a `.sha256` sidecar (Qwen3-ASR, Llama-3-ASR variants) trips this
allocation. On a 32 GB laptop resolving a 24 GB Qwen3-ASR split
across 4 shards, the verify step alone pushes the process over
16 GB resident.

## Recipe

Swap `std::fs::read` for a buffered `tokio::fs::File` +
`tokio::io::copy` into a streaming `Sha256` hasher
(`sha2::Digest::new()` accepts streaming updates). Concretely:

```rust
let mut file = tokio::fs::File::open(&target_path).await?;
let mut hasher = Sha256::new();
let mut buf = vec![0u8; 64 * 1024];
loop {
let n = file.read(&mut buf).await?;
if n == 0 { break; }
hasher.update(&buf[..n]);
}
```

Add a direct unit test for `verify_sha256_sidecars` that writes
a multi-MB target file, a sidecar with the correct hash, and
asserts the function returns `Ok`. A second test that tampers
with the sidecar to a wrong hash asserts the function returns
`Err` (today the function silently succeeds if no sidecar
exists; that contract should also be pinned by a test).

## Acceptance

- `verify_sha256_sidecars` streams the target file.
- Peak RSS during a Qwen3-ASR 24 GB resolve stays below
500 MB (regardless of shard size).
- New unit tests cover the happy path and the sidecar-tampered
path.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.