eduaguilera / eduaguilera/whep
Reconsider input-data hosting: move whep_inputs off the Nextcloud pins board
- Dominant language
- R
- Stars
- 1
- Forks
- 5
- Avg merge
- 1d 11h
- Merged PRs (30d)
- 183
Description
## Current setup
Input datasets referenced by `whep_inputs` are hosted on CSIC's Nextcloud (`saco.csic.es`) as a public WebDAV share, read via `pins::board_url()`. The read path (`R/input_files.R` / `whep_read_file()`) is: bundled local example board → remote `board_url` → stale local cache fallback. Publishing (`inst/scripts/prepare_upload.R`) is **manual**: build a versioned pin locally, upload the folder to Nextcloud by hand, hand-edit `_pins.yaml`, then update `whep_inputs.csv`.
The indirection (`whep_read_file` + `whep_inputs` + pins) is good and worth keeping — it decouples every caller from the storage backend, so the backend is swappable without touching pipeline code. The concerns are about *where the bytes live* and the *manual publish flow*, not pins itself.
## Weaknesses of the current hosting
- **Fragile URLs.** Every entry hardcodes a Nextcloud share token; if it rotates or storage is reorganized, all `board_url`s break at once — including for past "frozen" versions.
- **Uncertain bandwidth/uptime** for external users (institutional Nextcloud is not a CDN).
- **Manual, error-prone publishing** — the upload → edit `_pins.yaml` → edit CSV ritual is where version drift creeps in.
Note: the data behind `whep_inputs` is **mostly frequently-updated dev data**, not citable frozen artifacts — so Zenodo/DOI archival is out of scope here (could revisit only for a formal release snapshot).
## Proposed direction: GitHub Releases as a file store (via `piggyback`)
Treat GitHub release **assets** as a plain, free, CDN-backed file bucket (assets are arbitrary blobs stapled to a tag; nothing to do with the code tree).
- **Free, no credit card, no new cloud account** (unlike Cloudflare R2, which requires linking a payment method even for its free tier).
- Limits: **2 GiB per file, up to 1000 assets per release, no cap on total size or bandwidth.**
- Publishing becomes scripted: `piggyback::pb_upload()` / `pb_download()` (or `gh release upload`) replaces the manual Nextcloud steps.
- Reading barely changes: point `board_url` at stable release-asset URLs, or fetch assets directly; `whep_read_file` stays the single indirection point.
### Avoid conflating data with code releases
Put data in its **own repository** (e.g. `eduaguilera/whep-data`) whose releases hold only data — keeps the code repo's `v*` release history clean. (Alternative: dedicated data-only tags like `inputs-20260722` in this repo.) Recommendation: separate data repo.
### Versioning caveat
Releases don't give pins' version-*folders* for free. Version either by data-tag or by filename, recorded in `whep_inputs`. Acceptable for frequently-updated dev data, but it's a real difference from the current pins version semantics — decide the scheme explicitly.
## Large files and non-tabular data
- **Tabular (current parquet):** ideal fit. For files >2 GiB, use **partitioned parquet** (multiple part-files); `arrow` reads a directory of parts as one dataset natively (with predicate push-down) — cleaner than raw byte-splitting.
- **Rasters / NetCDF:** assets are format-agnostic. `whep_read_file` already supports non-tabular via its `raw`/`tar.gz` types (returns the file path) → hand to `terra`/`stars`/`ncdf4`. For large rasters, a **Cloud-Optimized GeoTIFF (COG)** can be read windowed straight off the release URL via GDAL `/vsicurl/` (release asset URLs support HTTP range requests) — an upgrade over the current host. Classic NetCDF has no efficient range read, so it's download-then-open (same as today; no regression). Server-side subsetting (OPeNDAP) is out of scope for any static-file host.
## When R2 would be worth the credit card instead
If a lot of data is GB-scale and needs partial/random access beyond COG, or if 1000-assets/2-GiB chunking gets unwieldy, Cloudflare R2 (native `pins::board_s3`, zero egress) becomes worth the payment-method friction. Otherwise GitHub Releases wins on zero setup cost.
## Open question to resolve before implementing
Actual file sizes and count in `whep_inputs` — confirms whether the 2 GiB/file limit and 1000-asset/release cap are ever a constraint in practice.
## Decision needed
1. GitHub Releases + piggyback (separate data repo) vs. R2 vs. status quo.
2. Versioning scheme (per-tag vs. per-filename) if moving to releases.
Related: #131 (validation-focused CLAUDE.md), #132 (renv removal).
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Contributor guide
Research direction
Start with R/input_files.R and whep_read_file(), then inspect inst/scripts/prepare_upload.R, whep_inputs.csv, and the current pins metadata. Inventory the actual file sizes and count, compare GitHub Releases with piggyback, R2, and the status quo, and resolve the repository and versioning choices before implementation; done means a documented decision and an agreed publish/read flow.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- github, r
- Domain
- cloud, data, release
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100