Generalize versioned dataset registries with verified multi-file bundle installation

Motivation: move shared machinery out of domain packages

We have two concrete implementations to consolidate:

• hitlist.VersionedDatasetRegistry already provides dataset/version resolution, pinned defaults, caller-supplied cache roots, and provenance. Its comment explicitly describes a reusable layer also consumed by tsarina. It currently models one file per dataset/version.
• mhcseqs's protein dataset module implements independently versioned records+manifest pairs and multi-file source bundles, with integrity validation, staging, publication, locking, and interrupted-swap recovery. These filesystem concerns now occupy a substantial part of a biological-data module.
• pyensembl.DownloadCache also has reference/annotation-version-specific cache layouts. Its single-file downloads are an immediate candidate for the lower-level public API; coordinated reference bundles are a possible later adopter, not an assumption that all existing pyensembl inputs have immutable upstream manifests.

Proposed scope

Build a small optional versioned-bundle API on #57 (verified downloads) and #58 (side-effect-free inspection). Start from hitlist's existing registry, rather than inventing another registry framework.

Consumer supplies dataset name, opaque data-version strings, a pinned default, cache root, and a mapping of relative asset names to URLs and optional/required integrity metadata as appropriate to the selected mode. For a verified immutable bundle, every asset must have trusted expected integrity metadata. Version labels are not Python package versions or mutable latest aliases.

Public operations should cover version/path resolution, read-only validation/status, and explicit installation/repair:

• Stage all required assets and verify the complete bundle before publishing it. An unsuccessful refresh leaves the previously usable bundle intact.
• Concurrent first installs converge on a verified installation. Overlapping forced installs serialize or safely reuse an equivalent winner; they must not strand a backup that blocks future repair.
• Library-managed readers must not observe a missing/mixed bundle during publication. Document the consistency guarantee and lifetime of returned paths rather than promising arbitrary external file readers an unlimited snapshot.
• Missing versus invalid versus recovery-required are explicit. Read-only cache reuse does not create locks or attempt recovery; installation/repair owns mutations.
• Keep independently versioned installations separate. Each installed bundle should carry enough provenance to validate it offline without another release/API lookup or a mutable cross-version manifest being the sole authority.
• Source installations and generated outputs have separate ownership/roots. Reinstalling an input bundle must not delete generated outputs. No generic library needs to orchestrate the biological rebuild itself.
• Validate asset paths stay inside the intended bundle; reject absolute/traversing paths and do not replace arbitrary directories containing unrelated user files.

Use one internal publication mechanism, not a public collection of lock/backup helpers that every caller must compose correctly. A plain data mapping is sufficient initially; schema parsing, remote hosting, biological annotations, and any application CLI remain with the consumer.

Acceptance tests / adoption proof

Use tiny offline fixtures and deterministic barriers/failure injection:

1. Install a single-file registry entry (hitlist shape), a records+manifest pair, and a multi-file source bundle (mhcseqs shapes).
2. Validate/load offline, including a valid read-only installation and a missing sibling version.
3. Fail the last asset download, integrity validation, and publication; preserve the old bundle in each case.
4. Exercise concurrent first installs, overlapping forced installs, and managed reads during publication; no mixed assets or permanently blocking backups.
5. Exercise interrupted-publication recovery through explicit installation/repair, without requiring network if a complete verifiable local copy is recoverable.
6. Demonstrate generated outputs outside the source root survive source reinstallation.

Document supported local-filesystem/platform semantics. Do not expand this into distributed coordination, a scheduler, cache garbage collection, or guarantees for every network filesystem. Reuse existing atomic/locking primitives where appropriate and keep the dependency footprint small.

Rollout

Implement the foundational download/inspection issues first, then this bundle API. Migrate mhcseqs and hitlist through separate focused PRs that delete duplicated code and preserve their public APIs/cache compatibility. Assess pyensembl adoption independently. Existing datacache users should not have to adopt bundles or a new registry format.

Downstream context: mhcseqs PR #81. Related project-boundary discussion: #26; this feature should not grow the database/ORM responsibilities of datacache.

Start by reading the foundational verified-download and side-effect-free inspection issues, #57 and #58, then study the linked hitlist VersionedDatasetRegistry and mhcseqs dataset module. Use tiny offline fixtures to cover single-file, paired, and multi-file installations, validation, failed publication, concurrency, and recovery. Done means the API preserves verified bundles and existing usable data while documenting supported filesystem semantics.