CosmoStat / CosmoStat/sp_validation

Migrate cosmo_val + Snakemake to the SACC writers

Open
#247 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
2
Forks
5
Avg merge
1d 14h
Merged PRs (30d)
19

Description

Make SACC the sole native output format of the cosmology-validation pipeline. Every two-point product (ξ± coarse and fine, pseudo-Cℓ, COSEBIs, pure-E/B, ρ/τ, n(z)) is written through the sacc_io builders into single-statistic SACC part files, and the Snakemake workflow assembles those parts into one terminal SACC file per catalogue version, {version}.sacc. No bespoke writer remains on the data-vector path.

Desired end state

  • Each cosmo_val statistic's write step is replaced by a call to the matching sacc_io builder, producing a part file that holds exactly one statistic plus the self-describing n(z) tracers it references. A producer writes a part with a single save(s, path) call; it never assembles a terminal file and never emits a bespoke format alongside.
  • A dedicated Snakemake assembly rule reads the parts, concatenates them in canonical order, attaches the BlockDiagonal covariance (analysis blocks plus the dense per-pair fine block), and writes the single terminal {version}.sacc. The terminal file is a pure gather; anything that wants only one statistic targets the corresponding part rule, not a second terminal file.
  • Part filenames are DAG internals: consumers bind parts through the workflow's path helpers, never by hardcoding names. {version}.sacc is the only stable name.
  • Blinding rides the DAG here. Blinding is per part, at birth (#241 §4; CLIs and the assembly-time assertion are delivered by #252): a blind-init rule fires once per catalogue version, each blindable part rule (coarse ξ±, fine ξ±, pseudo-Cℓ) is followed by blind-part so only blinded parts persist on disk, COSEBIs/pure-E/B are derived from the blinded fine ξ± (born blinded), and the terminal assembly asserts blind_commitment is identical across parts. This issue authors the rules; #252 supplies the machinery.
  • The bespoke .txt/custom-FITS/.npz writers on the data-vector path are deleted; nothing downstream reads them.

sacc_io is a fixed dependency and the single source of truth for every builder signature. Metadata keys required on every file: catalogue_version, sp_validation_version, created, npatch.

Acceptance

The workflow produces {version}.sacc per version from parts, with only blinded blindable parts persisted and the commitment assertion exercised; the deleted bespoke writers have no remaining readers; the assembled file matches the sacc_io layout contract.

— Fable on behalf of Cail.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by tracing the cosmo_val statistic write steps, the sacc_io builders, and the Snakemake workflow's path helpers. Map the blind-init, blind-part, and terminal assembly rules, then verify that all two-point products flow through single-statistic parts into one {version}.sacc file. Done means the bespoke writers have no readers, blindable parts persist only after blinding, and the assembly commitment assertion and sacc_io layout contract pass.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
build-system, data-engineering
Issue type
Refactor
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
28/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.