TimelyDataflow / TimelyDataflow/timely-dataflow

Tracking progress towards completion.

Open
#340 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Rust
Stars
3.6k
Forks
293
Avg merge
14h 46m
Merged PRs (30d)
4

Description

Is there a way to estimate "progress towards completion" in the context of either timely or differential? Let's say we pushed some updates to DD, advanced the timestamp and are now calling step_or_park in a loop waiting for the probe to report that all updates have successfully propagated through the pipeline. This can take a long time, e.g., when populating the pipeline with initial data or when processing a small update that triggers large amount of recomputation. In these situations it may be nice to comfort the user with a little progress bar showing approximately how much longer they have to wait.

@frankmcsherry, do you happen to have some magic up your sleeve that might help here? :)

@Kixiron, @RDambrosio016.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reading the timely and differential APIs around step_or_park and timestamp advancement. Determine whether the pipeline exposes enough information to estimate propagation or recomputation progress, then define what an approximate progress indicator should report for initial loads and small updates. Done would require an agreed design and an implementation path, but no files or tests are named.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
stream-processing
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.