TimelyDataflow / TimelyDataflow/timely-dataflow

Looping In Timely Dataflow

Open
#165 10 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

question
Dominant language
Rust
Stars
3.6k
Forks
293
Avg merge
14h 46m
Merged PRs (30d)
4

Description

let probe = worker.dataflow(|scope|{
                let (handle, cycle) = scope.loop_variable(loopNum, 1);
                scope.input_from(&mut input).exchange(|x| *x)
                                            .map(|x| work(&x, opVec[0]))
                                            .concat(&cycle)
                                            .map(|x| work(&x, opVec[2])) // loop body
                                            .connect_loop(handle);
                cycle.map(|x| (hash(&x), x)).probe(); // last line
});

Hello, in the code above I am trying to execute the loop body (commented above) n times using a loop then after I am done with the calculations I want to execute the last line. What is happening when I run it is that it executes the last line with the loop too, so both, loop body and last line, are looped. How can I fix this in such a way that after the whole loop is completed, I take the stream and I continue working on it?

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the loop_variable, connect_loop, and probe calls shown in the Rust snippet, then trace how the loop output is scheduled. The issue is done when the loop body runs the requested number of times and the final stream processing runs only after the loop completes.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
stream-processing
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.