scipp / scipp/ess

Streaming module: More fine-grained recomputes

Open
#123 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

essreduce
Dominant language
Python
Stars
1
Forks
3
Avg merge
2d 11h
Merged PRs (30d)
17

Description

This is a note to get it off my mind, currently not a requirement that needs to be worked on!

ess.reduce.streaming.StreamProcessor currently takes an all-or-nothing approach to updating and re-computing: It is assumed that all "dynamic inputs" are updated at the same frequency. Then, everything (non-static) is recomputed, accumulated, and final results computed.

In practice, there may be dynamic inputs that change with different frequency (for examples fast detector event data updates alongside slow detector bank move), and we may thus want to reflect this in the way the workflow updates/recomputes. Pre-computation of static inputs may be replaced by a more dynamic mechanism that can handle nodes that updated infrequently (or not at all).

As this is potentially more risky, since it seems to be constituting a potentially more error-prone caching mechanism, I think we should not perform such a change until a clear need is identified.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reading ess.reduce.streaming.StreamProcessor and tracing how dynamic inputs are updated, recomputed, and accumulated. This note has no implementation requirement; work is not ready to begin until a concrete need, update-frequency behavior, and correctness criteria are defined.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
stream-processing
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
15/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.