CosmoStat / CosmoStat/sp_validation
Migrate cosmo_val + Snakemake to the SACC writers
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 2
- Forks
- 5
- Avg merge
- 1d 14h
- Merged PRs (30d)
- 19
Description
Make SACC the sole native output format of the cosmology-validation pipeline. Every two-point product (ξ± coarse and fine, pseudo-Cℓ, COSEBIs, pure-E/B, ρ/τ, n(z)) is written through the sacc_io builders into single-statistic SACC part files, and the Snakemake workflow assembles those parts into one terminal SACC file per catalogue version, {version}.sacc. No bespoke writer remains on the data-vector path.
Desired end state
- Each
cosmo_valstatistic's write step is replaced by a call to the matchingsacc_iobuilder, producing a part file that holds exactly one statistic plus the self-describing n(z) tracers it references. A producer writes a part with a singlesave(s, path)call; it never assembles a terminal file and never emits a bespoke format alongside. - A dedicated Snakemake assembly rule reads the parts, concatenates them in canonical order, attaches the
BlockDiagonalcovariance (analysis blocks plus the dense per-pair fine block), and writes the single terminal{version}.sacc. The terminal file is a pure gather; anything that wants only one statistic targets the corresponding part rule, not a second terminal file. - Part filenames are DAG internals: consumers bind parts through the workflow's path helpers, never by hardcoding names.
{version}.saccis the only stable name. - Blinding rides the DAG here. Blinding is per part, at birth (#241 §4; CLIs and the assembly-time assertion are delivered by #252): a
blind-initrule fires once per catalogue version, each blindable part rule (coarse ξ±, fine ξ±, pseudo-Cℓ) is followed byblind-partso only blinded parts persist on disk, COSEBIs/pure-E/B are derived from the blinded fine ξ± (born blinded), and the terminal assembly assertsblind_commitmentis identical across parts. This issue authors the rules; #252 supplies the machinery. - The bespoke
.txt/custom-FITS/.npzwriters on the data-vector path are deleted; nothing downstream reads them.
sacc_io is a fixed dependency and the single source of truth for every builder signature. Metadata keys required on every file: catalogue_version, sp_validation_version, created, npatch.
Acceptance
The workflow produces {version}.sacc per version from parts, with only blinded blindable parts persisted and the commitment assertion exercised; the deleted bespoke writers have no remaining readers; the assembled file matches the sacc_io layout contract.
— Fable on behalf of Cail.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by tracing the cosmo_val statistic write steps, the sacc_io builders, and the Snakemake workflow's path helpers. Map the blind-init, blind-part, and terminal assembly rules, then verify that all two-point products flow through single-statistic parts into one {version}.sacc file. Done means the bespoke writers have no readers, blindable parts persist only after blinding, and the assembly commitment assertion and sacc_io layout contract pass.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- build-system, data-engineering
- Issue type
- Refactor
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 28/100