CosmoStat / CosmoStat/shapepipe

One ShapePipe workflow for data and image sims

Open
#891 1 comment 0 reactions 2 assignees View on GitHub

@martinkilbinger is already working on this.

Since Sep 12, 2026.

Dominant language
Python
Stars
18
Forks
14
Avg merge
8h 40m
Merged PRs (30d)
10

Description

The image-sims ShapePipe run is a separate orchestrator (example/cfis_image_sims plus the *_canfar_v2.0.bash dispatcher, driven from sp_validation's image_sims.smk), and the config cleanup that came with the data workflow (#847, #852) has broken it: its default.* symlinks dangle and it selects mask configs that no longer exist. Meanwhile the Snakemake workflow (#852) does everything the sims need — chunked ngmix, node-local vignette staging, completeness checks, resume, the run report — and the sims get none of it.

Target: sims are a branch of the one workflow, not a sibling.

  • rules/image_sims.smk in the main tree. Two config keys select the branch: kind: data|sims (where the pixels come from and how they're named) and psf_model: psfex|mccd|fake. They stay separate so sims can also run the real PSF chain once the star-injected grids exist.
  • The sims delta is small: a few _sims ini variants under workflow/config/cfis/, one new rule (tile_fake_psf), and tile_uncompress/exp_psf skipped for sims. Everything else is the data DAG unchanged.
  • Each shear branch (1z2z, 1p2z, …) is one campaign: five sp runs over the same sky with different input roots. sp_validation's sims workflow drives ShapePipe — its m-bias DAG calls sp run per branch, then reads only the campaign products (#879's final_cat_<campaign>.hdf5 plus a run manifest), never ShapePipe's run tree.
  • The workflow runs on both nibi and candide.
  • Gone: example/cfis_image_sims, the bash dispatcher (after a deprecation release — they're packaged entry points), and sp_validation's im_init/im_pipeline/im_merge with their ShapePipe-internal config keys.

Steps — each becomes a sub-issue when we're ready to pick it up:

  1. sp run --config-file and a candide profile (shapepipe). No dependencies; this is the two-machine work.
  2. #879 (in progress). The sims branch stacks on its final_cat_merge.
  3. The sims branch (shapepipe): kind/psf_model keys, _sims ini variants, tile_fake_psf, the two elided rules. Acceptance: one tile of grid_3 through sp run, compared against the existing catalogue.
  4. The boundary (sp_validation): run manifest, im_shapepipe calling sp run per branch, cat_config reading the manifest; im_init/im_pipeline/im_merge and their keys removed. The comprehensive catalogue comes from the v2 hdf5 readers (CosmoStat/sp_validation#343) rather than extract_info.py (CosmoStat/sp_validation#294).
  5. Deprecate example/cfis_image_sims and the bash dispatcher (shapepipe), one release later removal.

Supersedes the pipeline half of CosmoStat/sp_validation#284 / #285 / #287; related: #870, #885.

Rule table (data vs sims)
Rule data sims Parameterisation
tile_get_images ✓ ✓ config_tile_Git_{kind}.ini (INPUT_PATH, INPUT_FILE_PATTERN, INPUT_NUMBERING \d{3}\.\d{3} vs \d{3}-\d{3}, INPUT_FILE_EXT, OUTPUT_FILE_PATTERN). Plus a rule-side change: get_images_runner matches tile_numbers.txt, which unit_pre writes in dot format; sims need the dashed id, so unit_pre branches on kind (or exports a third var). Not ini-only.
tile_uncompress ✓ — Elided for sims (weights ship uncompressed); sims' Sx reads the weight from Git directly. Works because A's inis use fixed absolute INPUT_DIRs. Kills bit 2.
tile_find_exposures ✓ ✓ config_tile_Fe_{kind}.ini — not one ini. Verified diff: FILE_PATTERN CFIS_image→CFIS_simu_image, COLNUM 3→2, EXP_PREFIX p→simu_image-. Those two are FITS-HISTORY parsing facts, they define the tile↔exposure edge build_index.py consumes, and getting them wrong yields a wrong or empty exposure list silently.
exp_get_images ✓ ✓ config_exp_Gie_{kind}.ini (patterns + roots).
exp_split ✓ ✓ One ini; FILE_EXT → $SP_IMG_EXT. N_HDU = 40 identical.
exp_psf ✓ only if psf_model != fake Unchanged for data (post-#888). A sims × psfex variant is a fourth ini — see §2.1.
tile_fake_psf (new) — only if psf_model == fake Tile-level, between tile_detect and tile_vignets. Must join the tile_shape group — fake_psf_runner writes galaxy_psf*.sqlite, the store ngmix reads at random per object per epoch, which is exactly what node-local NVMe staging exists for; outside the group it lands on NFS. Joining means inheriting the group's resource contract, and NGMIX_VIGNET_DIR hardcodes run_sp_tile_PiViVi, so any sims PiViVi variant keeps that RUN_NAME.
tile_exp_forest, tile_merge_headers ✓ ✓ Unchanged (config_tile_Mh_exp_{kind}.ini for WARN_MISSING_EXP/INPUT_DIR).
tile_detect ✓ ✓ config_tile_Sx_{kind}.ini (weight source, image pattern).
tile_vignets ✓ ✓ config_tile_PiViVi_{psf_model}.ini gains a fake variant (vignetmakers only).
tile_ngmix ✓ ✓ Both keys: PSF-source dir on psf_model, background inputs + BKG_SUB on kind. Chunking, ngmix_ranges.json, the tile_shape group apply unchanged — sims gain parallel ngmix, their compute bottleneck, for free.
tile_merge_cats, tile_make_cat ✓ ✓ config_tile_Mc.ini carries no psf suffix on the data side; sims' config_tile_Mc_psfex.ini folds into it.
clean_exposure, clean_tile ✓ ✓ Unchanged.
final_cat_merge (#879) ✓ ✓ Already the campaign-merge rule; sims inherit it.

— Fable on behalf of Cail

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.