jejjohnson / jejjohnson/pipekit

pipekit-train: fsspec-backed `CachedDataset` (S3 / GCS / …) (v0.2)

Open
#18 0 comments 0 reactions 0 assignees View on GitHub
type:feature
Dominant language
Python
Stars
0
Forks
0
PR merge metrics
No merged PRs in 30d

Description

## Problem / Request

`CachedDataset` is documented as supporting fsspec URIs (`s3://`, `gcs://`, …) but the v0.1 implementation rejects non-local paths. Wire an fsspec-backed zarr store so cloud-cache works.

`boundaries.md` §13.1: "CachedDataset is local-filesystem only in v0.1. Fsspec URIs (s3://, gcs://) remain documented as the target, but the v0.1 implementation rejects them — fsspec-backed zarr stores land in v0.2 alongside the rest of the cache work."

## Motivation

- Cloud caching is the main reason to have a content-addressed cache — share cached simulation outputs across machines / runs.
- pipekit-experiment already does this for the model registry (`S3ModelRegistry`); same fsspec pattern.
- zarr supports fsspec-backed stores via `zarr.storage.FSStore` / `zarr.storage.RemoteStore`.

## Implementation Steps

- [ ] `CachedDataset.__init__` accepts `cache_dir` containing an fsspec URI; route through `fsspec.url_to_fs`.
- [ ] `_materialise` uses `zarr.storage.FSStore` (or zarr 3's `RemoteStore`) instead of local path.
- [ ] `_open_cache` mirrors.
- [ ] `invalidate` uses fsspec's filesystem API for deletion.
- [ ] Add `pipekit-train[s3]` extra mirroring `pipekit-experiment[s3]`.
- [ ] Tests: smoke test against in-memory fsspec filesystem (no real S3 dependency in CI).

## Definition of Done

- [ ] `CachedDataset(cache_dir="s3://bucket/path")` materialises to S3 and reads back.
- [ ] `s3` extra documented in pyproject + README.
- [ ] Docstring updated; §13.1 boundaries note resolved.

## Relationships

- Parent: pipekit-train v0.2
- Related: pipekit-experiment's S3ModelRegistry pattern.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.