scipp / scipp/ess

Tracking: migrate from sciline map/reduce to stages and aggregations (sciline ADR 0003)

Open
#746 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
1
Forks
3
Avg merge
2d 11h
Merged PRs (30d)
17

Description

Tracking issue for the ESS side of sciline ADR 0003, Replace map/reduce with stages composed outside the graph, proposed in https://github.com/scipp/sciline/pull/245. The rollout plan in the design document has the file-level sites for every item below, from a survey of the monorepo and esslivedata on 2026-09-11. Links point at the sciline branch until the ADR PR merges. Nothing here starts before the ADR is accepted and the additive sciline release is out.

In short: a Stage is the part of a pipeline from input keys to output keys with the static part held; an Aggregation is two stages with an accumulator per accumulation key between them. Each package returns its own object in place of a map/reduced pipeline, holding the pipeline, its aggregations, and the contributions by member.

ess.reduce, additive

  • Accumulators: maybe_hist moves out of Accumulator.push into the subclasses that histogram; a Forwarder; a test that every accumulator satisfies sciline.Accumulator.
  • StreamProcessor rewritten as a driver over three stages, a forwarder, and accumulators, keeping its interface, against the streaming, visualize, and accumulator tests. Needs the sciline follow-up (provide, reporter, visualize support).
  • assign_parameter_values and parameter_mappers replaced by the package-object protocol, with or after esssans.
  • docs/user-guide/polarization/zoom.ipynb (uses get_mapped_node_names and with_sample_runs) migrates with esssans.

Reduction packages, one PR each, in this order

  • esssans: with_sample_runs/with_background_runs become one aggregation per run type with a shared finalize stage held by a package object; with_pixel_mask_filenames becomes a PixelMaskFilenames list parameter and two providers; with_banks becomes compute_members; ZoomTransmissionFractionWorkflow; parameter_mappers registrations; LoKI and zoom notebooks; tests using compute_mapped. Sets the package-object pattern.
  • essdiffraction: with_pixel_mask_filenames becomes a list parameter and one provider and loses the empty-list cyclebane workaround. All 22 test call sites pass [], so add one test with a mask file. The two DREAM advanced-notebook tables (one with two columns) become an aggregation.
  • essspectroscopy: three bank folds in the bifrost builders. Open design point for that PR: NeXusData depends on the run, so the bank aggregation sits inside per-run work when runs are members; either the package object owns a per-run bank aggregation or a DetectorNames list parameter with a looping provider. compute_mapped in one test and one notebook becomes compute_members.
  • essreflectometry: with_filenames becomes an aggregation with up to seven accumulation keys and loses the try/except around each reduce; BatchProcessor.compute loses its compute_mapped fallback; batch_processor enters the aggregation for a list-valued Filename[SampleRun]; the GUI folds; amor and estia notebooks; tests.
  • essnmx: notebooks and one test fixture only; mcstas_workflow.ipynb imports cyclebane.graph directly and must stop.
  • essimaging: drop visualize(compact=) in two notebooks.

Every package also drops visualize(compact=) from its notebooks in its own PR. constraints= (essreduce nexus workflows, offspec, amor) stays until the breaking sciline release.

esslivedata

  • The bifrost bank fold in config/instruments/bifrost/factories.py sits in the static part of the streaming pipeline; it becomes Aggregation(...).compute(...) with the result set as a parameter before the StreamProcessor is built. Consumes the rewritten StreamProcessor through the essreduce bump.
  • Drop the direct cyclebane>=26.9.0 pin once sciline drops cyclebane.

Then

  • essreduce raises its sciline minimum to the breaking release; offspec and amor drop constraints= in the same bump.
  • essapps: apply the edits listed at the end of stages.md on branch architecture-sketch, reading Fold as Aggregation and "accumulation point" as accumulation key.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reading docs/developer/adr/0003-replace-map-reduce-with-stages-outside-the-graph.md and docs/developer/architecture-and-design/map-reduce-outside-the-graph.md, after the ADR and additive sciline release are available. Use the package-specific rollout items to locate the affected tests, notebooks, StreamProcessor, and config/instruments/bifrost/factories.py; done means the listed migrations are complete and the referenced tests and notebooks pass.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data-engineering
Issue type
Refactor
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.