Performance redesign: ZIP parity with transactional guarantees

Summary and experiment links

Follow-up to #310 (currently closed). Redesign reVault's read/write architecture to improve small-file, large-file, streaming and random-access performance while preserving transactional guarantees. This issue records the agreed plan; it does not declare the experiment successful or authorize default activation.

• Experimental branch: issue-310-zip-read-performance-rebased
• Preserved experimental snapshot: 95ad308
• Three-way benchmark report
• Raw benchmark results
• All case/profile summaries
• Experimental correctness audit

The benchmark binaries use source revision 6ed6cf08dbe5da217c34bb0677b3c969d5ef35ed. The existing-format column is the current improved performance branch without native-block-layout, not an old main-branch executable. The experimental column uses the same source with that feature enabled. Subsequent fault-sweep test edits do not affect those binaries.

Snapshot validation: the expanded storage fault sweep passes all 16 protection/compression/padding combinations for both default and experimental writers (24 injected boundaries per combination, 384 per writer). Strict native-feature library/test Clippy passes. Formatting ran last through the commit hook. These focused passes do not establish a green complete experimental suite or ZIP parity.

Why a redesign review is necessary

Interactive full-streaming times, milliseconds for the entire workload; lower is better:

Workload	ZIP	Existing format	Experimental
512 x 4 KiB, raw	6.50	3.78	18.69
512 x 4 KiB, compressed	6.97	3.26	20.55
4 x 1 MiB, compressed	0.88	1.69	5.24
4 x 8 MiB, raw	8.53	48.80	22.04
4 x 8 MiB, compressed	7.36	11.57	12.08
4 x 64 MiB, raw	76.07	398.38	178.07
4 x 64 MiB, compressed	59.03	99.51	320.91

Three 8 KiB ranges per file across four raw 8 MiB files: ZIP 0.106 ms, existing 46.49 ms, experimental 0.964 ms. The experiment improves raw partial reads dramatically but still misses ZIP and sacrifices existing small-file and compressed-read wins.

Method: unsigned/unencrypted, default size padding, fresh file-backed handles, warm filesystem caches, nine samples per run, existing/experimental/experimental/existing order on one CPU. All 20 processes and 4,032 timed operations completed with byte assertions. Cold storage, encrypted/signed performance and write throughput are not established by this comparison. See the report for timing spread, codec differences, archive sizes, open times and partial-read integrity differences.

Small-file open times also regress. The last full experimental unit run had 15 failures; the subsequently expanded fault sweep passes, but the complete suite must be rerun rather than inferring a new failure count. Broad ZIP parity is not achieved.

Agreed objectives and hard constraints

• Keep pure Rust and one common archive architecture across encryption/signing modes.
• Permit a migration-first format replacement. Do not preserve the experimental format simply because implementation effort has already been spent on it.
• For signed archives, authenticate committed index metadata at open and verify content before returning it; provide explicit full-archive verification. Opening successfully will no longer imply that every untouched payload has been checked. This semantic change must be documented in APIs and bindings.
• Preserve cryptographic integrity and authentication. Unsigned/unencrypted checksums detect corruption, not malicious rewriting by an attacker who can replace the whole archive; do not imply owner authentication without signing.
• Preserve atomic publication, durability, rollback, crash recovery, resumable cleanup and secure deletion. These guarantees are not negotiable performance trade-offs.
• Target ZIP parity for unsigned/unencrypted reads. Compare protected modes with equivalent existing reVault modes; do not claim security-equivalent ZIP parity where it was not measured.
• Allow up to 10% timing trade-offs for writes or selected reads only with substantial measured benefits. Retain the earlier 5% open/load regression limit. These are release ceilings, not optimization targets; report every regression rather than averaging it away.
• Default codec, layout and padding choices may be reconsidered, but each security/privacy/compatibility default change requires explicit approval. Never silently remove padding from existing archives.

Phase 1: establish causes before further format integration

• Freeze production implementation while completing a bounded architecture/profile review. Preserve the current existing-format and experimental binaries as baselines.
• Profile small-file reads and compressed 1 MiB/64 MiB reads, where the experiment regresses most.
• Separate open, index/manifest parsing, storage reads, hashing, decompression, allocation, copying and wiping.
• Add diagnostic counters for bytes fetched/decoded/copied/wiped, metadata decodes, cache hits and frame decompressions. Exclude instrumentation overhead from acceptance timings.
• Verify frame sizes and compression decisions for identical logical inputs.
• Explain each major regression with a profile and focused reproduction before proposing its fix.

Code inspection confirms that native reads bypass the existing shared frame cache and reopen/decode page metadata per chunk. These are investigation targets, not a complete causal explanation of the size-dependent compressed regressions.

Phase 2: shared read engine and common indexed format

Shared read engine

• Use one verified read engine for streaming, extraction, range reads and file handles.
• Share authenticated metadata and decoded frames across handles within a bounded archive cache. Key cached state by archive identity, committed generation and immutable frame/block identity, not offsets alone.
• Pack small files and reuse verified shared metadata/data rather than reopening and decoding the shared frame for every file.
• Support direct raw reads into caller buffers after verifying complete touched integrity units. Never expose unverified bytes on failure.
• Stream large files through bounded buffers, avoiding repeated full-frame allocation, copies and decoder initialization.
• Preserve source-change checks, failure latching and sensitive-buffer wiping. Alternative buffer strategies must pass the same security tests.

Format

• Separate compression frames, independently verifiable blocks and physical padding/allocation.
• Authenticate a root covering file mappings, block commitments and transaction-critical metadata. Bind block identity, ordering, lengths, codec and archive context to verification.
• Keep compression and protection independent, without a separate plaintext-only format.
• Keep recovery metadata discoverable, but never treat stale/uncommitted manifests as proof of the latest committed state.
• Support explicit full-content verification without scanning every payload during normal open.

Compression and defaults

• Benchmark single-pass Zstd configurations, compression-frame sizes and verification-block sizes independently.
• Investigate decoder workspace reuse only where profiles justify it; do not assume previously rejected experiments now work.
• Consider alternative pure-Rust codecs and cryptographic implementations, subject to compatibility and security review.
• Keep second-pass encoding experimental unless its measured write cost fits the accepted budget.
• Measure padded and unpadded archives separately and present any proposed default changes explicitly.

Retain candidates only after correctness and performance gates pass. Rank passing candidates by end-to-end read performance, then write time and archive size, not isolated codec microbenchmarks.

Phase 3: transactional and recovery proof

The user's explicit instruction: do not compromise the transactional nature of updates or the system's ability to rollback/recover.

• Before publication, failures leave the previous committed state intact or recoverable. Staged work must not overwrite live committed allocations.
• After publication, recovery rolls forward to the published state, never exposes a mixed state and never rolls back to obsolete metadata.
• Preserve durable preparation records, authenticated publication ordering and resumable cleanup checkpoints.
• Authenticate allocation ownership before erase, reuse or truncation. Verify-on-access must not defer validation needed for safe destructive recovery operations.
• Preserve poisoned-handle refusal, nonce uniqueness on retries and retirement/cache invalidation.
• Expand fault injection across creation, import, replacement, deletion, rename, compaction and migration, including operations that write before commit().
• Exercise every write, append, sync, publication, cleanup and truncation boundary; torn writes, interrupted/repeated recovery, corrupted indexes, stale pages and external-source changes.
• Test all encryption/signing/compression/padding combinations. Verify persisted bytes through independent reopen/read operations.
• Use public CLI lifecycles where available; document internal mutation exceptions for corruption and storage-fault injection.

A candidate that weakens any guarantee is rejected regardless of speed.

Phase 4: performance acceptance

• Compare ZIP, existing reVault and each candidate on identical small/many-file, 1 MiB, 8 MiB, 64 MiB and larger streaming workloads.
• Include repetitive, incompressible and realistic mixed content; sequential reads, shuffled access, small ranges, extraction and repeated reads.
• Separate fresh handles, warm-handle caches and controlled cold-storage cases.
• Measure open/load, write/encrypt/sign, mutation/commit, peak memory and archive size.
• Use alternating paired runs, fixed sample policies and uncertainty estimates; repeat unstable cases separately.
• Keep all workload regressions visible and preserve the original ZIP comparison rather than weakening it to manufacture parity.

No default activation until correctness/transaction/recovery suites pass; every applicable workload satisfies the read/write and load limits; ZIP parity and any remaining gaps are reported explicitly; and security/privacy/compatibility default changes are separately approved.

Phase 5: migration, public interfaces and delivery

• Inventory released formats and local lockboxes before assigning a new unambiguous format identifier.
• Provide explicit non-destructive migration to a separate destination; verify content and metadata before replacement. Interrupted migration must leave the source usable.
• Give older archives clear migration guidance, without silent rewriting. Detect earlier experimental variants safely rather than reinterpreting incompatible bytes under the same identifier.
• Expose persistent creation choices and full-verification behavior consistently through Rust, CLI and language bindings.
• Update migration tools, conformance tests and changelogs together.

Delivery gates: diagnostic review -> minimal read-engine prototype -> performance decision -> transactional integration -> migration/bindings -> release review.

Do not resume broad format integration before the prototype demonstrates that it preserves the existing reader's wins. Do not mark this issue implemented/tested/reviewed merely because an experimental branch exists or focused tests pass.

Start with the benchmark report and audit files under rust/revault_lockbox_api/benches/results/, then inspect the native-read path and shared frame-cache behavior identified in the issue. Compare the preserved experimental snapshot with the existing-format baseline before proposing changes. Done requires passing correctness, transaction, recovery and performance gates, with ZIP parity and remaining gaps reported explicitly.