Tracking: migrate from sciline map/reduce to stages and aggregations (sciline ADR 0003)
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 1
- Forks
- 3
- Avg merge
- 2d 11h
- Merged PRs (30d)
- 17
Description
Tracking issue for the ESS side of sciline ADR 0003, Replace map/reduce with stages composed outside the graph, proposed in https://github.com/scipp/sciline/pull/245. The rollout plan in the design document has the file-level sites for every item below, from a survey of the monorepo and esslivedata on 2026-09-11. Links point at the sciline branch until the ADR PR merges. Nothing here starts before the ADR is accepted and the additive sciline release is out.
In short: a Stage is the part of a pipeline from input keys to output keys with the static part held; an Aggregation is two stages with an accumulator per accumulation key between them. Each package returns its own object in place of a map/reduced pipeline, holding the pipeline, its aggregations, and the contributions by member.
ess.reduce, additive
- Accumulators:
maybe_histmoves out ofAccumulator.pushinto the subclasses that histogram; aForwarder; a test that every accumulator satisfiessciline.Accumulator. -
StreamProcessorrewritten as a driver over three stages, a forwarder, and accumulators, keeping its interface, against the streaming, visualize, and accumulator tests. Needs the sciline follow-up (provide, reporter, visualize support). -
assign_parameter_valuesandparameter_mappersreplaced by the package-object protocol, with or after esssans. -
docs/user-guide/polarization/zoom.ipynb(usesget_mapped_node_namesandwith_sample_runs) migrates with esssans.
Reduction packages, one PR each, in this order
- esssans:
with_sample_runs/with_background_runsbecome one aggregation per run type with a shared finalize stage held by a package object;with_pixel_mask_filenamesbecomes aPixelMaskFilenameslist parameter and two providers;with_banksbecomescompute_members;ZoomTransmissionFractionWorkflow;parameter_mappersregistrations; LoKI and zoom notebooks; tests usingcompute_mapped. Sets the package-object pattern. - essdiffraction:
with_pixel_mask_filenamesbecomes a list parameter and one provider and loses the empty-list cyclebane workaround. All 22 test call sites pass[], so add one test with a mask file. The two DREAM advanced-notebook tables (one with two columns) become an aggregation. - essspectroscopy: three bank folds in the bifrost builders. Open design point for that PR:
NeXusDatadepends on the run, so the bank aggregation sits inside per-run work when runs are members; either the package object owns a per-run bank aggregation or aDetectorNameslist parameter with a looping provider.compute_mappedin one test and one notebook becomescompute_members. - essreflectometry:
with_filenamesbecomes an aggregation with up to seven accumulation keys and loses thetry/exceptaround each reduce;BatchProcessor.computeloses itscompute_mappedfallback;batch_processorenters the aggregation for a list-valuedFilename[SampleRun]; the GUI folds; amor and estia notebooks; tests. - essnmx: notebooks and one test fixture only;
mcstas_workflow.ipynbimportscyclebane.graphdirectly and must stop. - essimaging: drop
visualize(compact=)in two notebooks.
Every package also drops visualize(compact=) from its notebooks in its own PR. constraints= (essreduce nexus workflows, offspec, amor) stays until the breaking sciline release.
esslivedata
- The bifrost bank fold in
config/instruments/bifrost/factories.pysits in the static part of the streaming pipeline; it becomesAggregation(...).compute(...)with the result set as a parameter before theStreamProcessoris built. Consumes the rewrittenStreamProcessorthrough the essreduce bump. - Drop the direct
cyclebane>=26.9.0pin once sciline drops cyclebane.
Then
- essreduce raises its sciline minimum to the breaking release; offspec and amor drop
constraints=in the same bump. - essapps: apply the edits listed at the end of
stages.mdon brancharchitecture-sketch, readingFoldasAggregationand "accumulation point" as accumulation key.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reading docs/developer/adr/0003-replace-map-reduce-with-stages-outside-the-graph.md and docs/developer/architecture-and-design/map-reduce-outside-the-graph.md, after the ADR and additive sciline release are available. Use the package-specific rollout items to locate the affected tests, notebooks, StreamProcessor, and config/instruments/bifrost/factories.py; done means the listed migrations are complete and the referenced tests and notebooks pass.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data-engineering
- Issue type
- Refactor
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100