openvax / openvax/datacache

Support integrity-safe resumable HTTP downloads

Open
#64 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
5
Forks
7
Avg merge
2h 44m
Merged PRs (30d)
3

Description

Motivation

Current retry support deliberately starts every HTTP attempt from byte zero and removes the failed staging file. That is safe for ordinary artifacts, but expensive for multi-gigabyte immutable releases. eureka-bench has a checksum-pinned CatPred archive of 10,207,582,747 bytes; losing a connection near completion should not require another full transfer.

Downstream context: https://github.com/iskandr/eureka-bench/pull/254 and https://github.com/iskandr/eureka-bench/issues/255.

Proposed scope

Add an explicit opt-in resume mode for raw HTTP/HTTPS downloads. Preserve a stable private partial beside the destination, issue a Range request from its current size, and publish only after the complete installed bytes satisfy expected_size and expected_sha256.

Required safety behavior

  • Require trusted integrity metadata for persistent/resumable partials, ideally SHA-256 plus size.
  • Preserve an already valid destination throughout download, retry, validation, and publication.
  • Validate 206 Content-Range starts at the requested offset; if the server returns 200 or an incompatible range, safely restart from zero.
  • Record and enforce a strong validator when available (ETag and/or Last-Modified); discard a partial if remote identity changed.
  • Handle a partial larger than expected, a complete-but-corrupt partial, 416, truncated streams, and servers without range support.
  • Keep partials owner-only, constrain them to the destination directory, and never follow an unsafe partial-file type.
  • Make retry/backoff continue from the last safely written offset.
  • Report cumulative bytes and total size through the progress callback without double-counting retries.
  • Allow callers to discard partials explicitly and document that resumable mode intentionally leaves bounded staging state after interruption.

Acceptance tests

Use a local HTTP fixture supporting Range, validators, dropped connections, ignored ranges, wrong Content-Range, and remote replacement. Test successful resume, safe restart, final integrity failure, Ctrl-C preservation, concurrent callers, and unchanged non-resumable behavior.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reading the existing HTTP download and retry path, then run the local HTTP fixture tests described in the issue. Done means opt-in downloads safely resume with trusted metadata, preserve valid destinations, handle the listed range and interruption cases, report progress correctly, and leave non-resumable behavior unchanged.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.