jejjohnson / jejjohnson/pipekit
pipekit-train: fsspec-backed `CachedDataset` (S3 / GCS / …) (v0.2)
- Dominant language
- Python
- Stars
- 0
- Forks
- 0
- PR merge metrics
- No merged PRs in 30d
Description
## Problem / Request
`CachedDataset` is documented as supporting fsspec URIs (`s3://`, `gcs://`, …) but the v0.1 implementation rejects non-local paths. Wire an fsspec-backed zarr store so cloud-cache works.
`boundaries.md` §13.1: "CachedDataset is local-filesystem only in v0.1. Fsspec URIs (s3://, gcs://) remain documented as the target, but the v0.1 implementation rejects them — fsspec-backed zarr stores land in v0.2 alongside the rest of the cache work."
## Motivation
- Cloud caching is the main reason to have a content-addressed cache — share cached simulation outputs across machines / runs.
- pipekit-experiment already does this for the model registry (`S3ModelRegistry`); same fsspec pattern.
- zarr supports fsspec-backed stores via `zarr.storage.FSStore` / `zarr.storage.RemoteStore`.
## Implementation Steps
- [ ] `CachedDataset.__init__` accepts `cache_dir` containing an fsspec URI; route through `fsspec.url_to_fs`.
- [ ] `_materialise` uses `zarr.storage.FSStore` (or zarr 3's `RemoteStore`) instead of local path.
- [ ] `_open_cache` mirrors.
- [ ] `invalidate` uses fsspec's filesystem API for deletion.
- [ ] Add `pipekit-train[s3]` extra mirroring `pipekit-experiment[s3]`.
- [ ] Tests: smoke test against in-memory fsspec filesystem (no real S3 dependency in CI).
## Definition of Done
- [ ] `CachedDataset(cache_dir="s3://bucket/path")` materialises to S3 and reads back.
- [ ] `s3` extra documented in pyproject + README.
- [ ] Docstring updated; §13.1 boundaries note resolved.
## Relationships
- Parent: pipekit-train v0.2
- Related: pipekit-experiment's S3ModelRegistry pattern.
Contributor guide
Assessment
This issue has not been assessed yet.