TimelyDataflow / TimelyDataflow/timely-dataflow
Tracking progress towards completion.
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 3.6k
- Forks
- 293
- Avg merge
- 14h 46m
- Merged PRs (30d)
- 4
Description
Is there a way to estimate "progress towards completion" in the context of either timely or differential? Let's say we pushed some updates to DD, advanced the timestamp and are now calling step_or_park in a loop waiting for the probe to report that all updates have successfully propagated through the pipeline. This can take a long time, e.g., when populating the pipeline with initial data or when processing a small update that triggers large amount of recomputation. In these situations it may be nice to comfort the user with a little progress bar showing approximately how much longer they have to wait.
@frankmcsherry, do you happen to have some magic up your sleeve that might help here? :)
@Kixiron, @RDambrosio016.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reading the timely and differential APIs around step_or_park and timestamp advancement. Determine whether the pipeline exposes enough information to estimate propagation or recomputation progress, then define what an approximate progress indicator should report for initial loads and small updates. Done would require an agreed design and an implementation path, but no files or tests are named.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- stream-processing
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 20/100