Feature Request: stream-hash remote asset downloads to avoid a second full file pass
- Dominant language
- Python
- Stars
- 133k
- Forks
- 15.7k
- Avg merge
- 1d 7h
- Merged PRs (30d)
- 158
Description
## Feature Idea
Stream-hash remote asset downloads, or add a core path that lets `upload_from_temp_path(...)` reuse a precomputed hash.
Right now the remote asset import flow appears to do this:
1. Download the remote file into a temp/staging file.
2. Call `upload_from_temp_path(...)`.
3. `upload_from_temp_path(...)` runs `compute_blake3_hash(temp_path)` before ingesting/registering the asset.
That means large remote model imports pay for:
- one full write of the downloaded file
- then one full read of the same file again just to compute the hash
On slower storage this becomes very noticeable. In our case, a `6.9 GB` safetensors import spent several extra minutes after the file had already finished downloading and landed on disk. The issue is not specific to our environment; it should affect any setup where the asset temp directory is on slower/networked/external storage.
A core improvement here would be either:
- hash the file incrementally while streaming the HTTP response into the temp file, then pass that hash into the ingest path
- or extend the ingest/upload API so callers can provide a trusted precomputed `blake3` digest for the just-downloaded temp file
The nice property is that this does **not** need to change the dedupe semantics. The asset hash is still the same `blake3`, but remote downloads would no longer need a second full-disk pass before registration.
A shape like this would already help a lot:
- keep the existing `upload_from_temp_path(...)` behavior as the default
- add optional support for a precomputed `asset_hash` / `digest`
- if not provided, fall back to the current `compute_blake3_hash(temp_path)` path
That would let remote download flows stay fast without forcing a breaking change on all existing callers.
## Existing Solutions
I did not find an existing core issue for this specific assets-path optimization.
I also did not find an existing core API that allows remote download code to reuse a streamed `blake3` result during `upload_from_temp_path(...)`.
Contributor guide
Assessment
This issue has not been assessed yet.