imazen / imazen/codec-corpus

R2Corpus: anonymous-pull, auth-push corpus sync with tar bundling

Open
#2 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
13
Forks
0
PR merge metrics
No merged PRs in 30d

Description

## Summary

Add an `R2Corpus` API to codec-corpus for syncing file corpora (fuzz seeds, test fixtures, conformance suites) to/from Cloudflare R2. Anonymous pull via public HTTP, authenticated push via S3 API.

The existing `Corpus` API (GitHub release tarballs) is widely used (~115 callsites across zen crates) and works well for curated, versioned datasets. `ManifestCorpus` (JSONL + per-blob fetch from R2) was never adopted outside this crate — its schema was overdesigned and the blob-by-hash mental model didn't match how people think about test data.

`R2Corpus` targets the gap: **continuously-growing corpora** (fuzz seeds, CI-generated test data) that are too large or too frequently updated for GitHub releases, but still need anonymous cross-platform fetch.

## Design principles

1. **Path-keyed, not hash-keyed.** Files have meaningful paths. The API returns a directory you walk with `std::fs`.
2. **Manifest is auto-generated, never hand-edited.** The `push` command regenerates the `.list` file. No manual schema maintenance.
3. **Minimal schema.** `.list` contains `{path: {size, sha256, in_bundle}}`. No `expected_behavior`, no `source_label`, no classification metadata.
4. **Smart bundling.** For prefixes with many small files, a `.tar.zst` bundle + `.list` sidecar replaces thousands of individual HTTP requests.
5. **Hierarchical prefixes.** `pull("fuzz/zentiff/")` fetches a subtree. `pull("fuzz/")` fetches everything.
6. **Same mental model as `Corpus`.** One-line setup, returns a `PathBuf`, users walk it normally.

## Storage layout

```
s3://codec-corpus/fuzz/zentiff/fuzz_decode.list # auto-generated index
s3://codec-corpus/fuzz/zentiff/fuzz_decode.tar.zst # frozen snapshot bundle
s3://codec-corpus/fuzz/zentiff/fuzz_decode/abc123... # individual delta (added since last bundle)
s3://codec-corpus/fuzz/zentiff/fuzz_decode/def456... # individual delta
```

## `.list` format

```json
{
"version": 1,
"prefix": "fuzz/zentiff/fuzz_decode/",
"generated_at": "2026-04-08T03:15:00Z",
"bundle": {
"key": "fuzz/zentiff/fuzz_decode.tar.zst",
"size": 18234000,
"sha256": "0123abc...",
"file_count": 1556,
"uncompressed_size": 41499000
},
"files": {
"abc123...": { "size": 1234, "sha256": "abc123...", "in_bundle": true },
"fed987...": { "size": 512, "sha256": "fed987...", "in_bundle": false }
}
}
```

## Pull algorithm

1. Fetch `.list` (small, one request)
2. Diff against local cache (by sha256)
3. If >40% of bundled files are missing → download `.tar.zst`, extract (1-2 requests for cold start)
4. Fetch remaining individual deltas in parallel (for incremental updates)

Cold start for 1556 files / 41 MB: ~3 seconds (1 GET for list + 1 GET for bundle + extract) vs ~30-60s for 1556 individual GETs.

## Push algorithm

1. Walk local dir, hash new files
2. Upload new files as individual objects + update `.list`
3. Optionally `--rebundle` to regenerate `.tar.zst` from all files (when too many deltas accumulate)
4. Auto-rebundle threshold (e.g., rebundle when >100 deltas exist since last bundle)

## HTTP client approach

**Note:** The existing `Corpus` API deliberately avoids compiled-in HTTP dependencies. It shells out to OS tools (`curl`, `wget`, `powershell Invoke-WebRequest`, `git sparse-checkout`) detected at runtime, keeping the crate tiny (~1 KB on crates.io) with minimal deps (`dirs`, `fd-lock`).

`R2Corpus` should follow the same pattern for **anonymous pull**:
- Bundle download: `curl -fSL` / `wget -q` / `Invoke-WebRequest` (same as existing `Corpus`)
- `.list` download: same tool chain
- Tar extraction: `tar` (with `zstd` support, i.e. `tar --zstd -xf` or `zstd -d | tar -xf`)
- Individual file fetch: `curl`/`wget` with parallel invocations (e.g. `xargs -P`)

For **authenticated push**, the S3 API is more complex (SigV4 signing, multipart upload). Options:
- Shell out to `aws s3` CLI (simple, user installs it themselves, push is admin-only anyway)
- Optional `r2-push` feature flag that pulls in `object_store` or `rust-s3` (heavier, but self-contained)
- Separate `r2-corpus-admin` binary with its own Cargo.toml and heavier deps (keeps the library crate minimal)

The **library crate** (`codec-corpus`) stays minimal with OS-tool downloads. The **admin binary** (push/rebundle) can afford heavier deps since it's only used by maintainers.

## Rust API

```rust
// Anonymous pull (no auth needed, R2 bucket has public-read)
// Uses curl/wget/powershell under the hood — no compiled-in HTTP client
let corpus = R2Corpus::pull("https://corpus.imazen.io", "fuzz/zentiff/fuzz_decode/")?;
for entry in std::fs::read_dir(corpus.path())? { /* ... */ }

// With options
let corpus = R2Corpus::pull_with_options(
"https://corpus.imazen.io",
"fuzz/zentiff/fuzz_decode/",
PullOptions { mode: PullMode::ForceBundle, ..default() },
)?;
```

## CLI (`r2-corpus` binary)

```bash
r2-corpus pull fuzz/zentiff/ # anonymous, downloads subtree
r2-corpus push fuzz/zentiff/fuzz_decode/ --local ./fuzz/corpus/fuzz_decode # auth required
r2-corpus push fuzz/zentiff/fuzz_decode/ --rebundle # regenerate tar.zst
r2-corpus list fuzz/zentiff/ # show .list contents
r2-corpus diff fuzz/zentiff/fuzz_decode/ # local vs remote
r2-corpus login # store R2 credentials
```

## Per-project config

Projects configure their corpus sync in `corpus-sync.toml`:

```toml
[corpus]
base_url = "https://corpus.imazen.io"
local_dir = "fuzz/corpus"

[[sync]]
prefix = "fuzz/zentiff/fuzz_decode/"
local = "fuzz/corpus/fuzz_decode"
```

## Dependencies

**Library crate (anonymous pull):** No new compiled deps — shells out to `curl`/`wget`/`tar`/`zstd` like the existing `Corpus` API. Keeps the crate tiny.

**Admin binary (push/rebundle):** Either shells out to `aws s3` CLI or uses optional compiled deps:
- `object_store` or `rust-s3` for S3-compatible push
- `serde` + `serde_json` for `.list` generation (already a dep via `ManifestCorpus`)
- `clap` for CLI
- `dirs` for cross-platform cache directory (already a dep)

## Implementation plan

**Phase 1 (~2 days):** Basic `R2Corpus` with per-file pull/push and auto-generated `.list`. No bundle support. Proves the API works.

**Phase 2 (~3 days):** Add `.tar.zst` bundle support. Smart pull heuristic. Rebundle command. Transparent upgrade — existing prefixes get faster as soon as someone runs `--rebundle`.

## Prerequisites

- [ ] Enable public-read on `codec-corpus` R2 bucket (or the `fuzz/` prefix)
- [ ] Optionally set up custom domain (`corpus.imazen.io` → R2 bucket)

## Motivation

This replaces the ad-hoc `fuzz-corpus.sh` shell script currently in `imazen/zenextras` (which shells out to `aws s3 sync` and requires AWS CLI + R2 credentials even for reads). Any zen crate could adopt this with `codec-corpus = { version = "1.1", features = ["r2"] }` and a one-line pull call.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reading the existing `Corpus` API and the `imazen/zenextras` `fuzz-corpus.sh` script to understand current corpus syncing. Then resolve the open library/admin boundary and implement the planned pull, push, manifest, and bundling phases; done means the documented Rust API and `r2-corpus` commands support the stated sync workflow.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, rust
Domain
backend, cli, cloud
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.