TimelyDataflow / TimelyDataflow/differential-dataflow

lots of questions about df-dataflow doc

Open
#127 5 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

question
Dominant language
Rust
Stars
3k
Forks
211
Avg merge
10h 42m
Merged PRs (30d)
34

Description

i have asked in gitter,im ..it is a really hard to understand.
so i wrote my questions.

i read the http://www.frankmcsherry.org/differential-dataflow doc.
and i cant figure out how every step acts,and why and when to use these function.
Through chapter 1,get started. i can copy the run and run ,but ,every step is not that explained enough. so ,when i finished the getting started,i'm crazy...

and the doc is not related in examples in github repo...

then i Found chapter 4, which explained a little more detail.

so ,i will wrote my question with chapter 4.
Any timely dataflow stream of the correct record time, specifically (data, time, diff), can be re-interpreted as a differential dataflow collection using the AsCollection trait, which provides a method as_collection()

what the hell is (data, time,diff)? where do they come from?

The InputSession type provides a method advance_to(time), which moves the internal time of the session forward to time, and prevents you from supplying input changes at times that are not greater or equal to time. This is a very strong statement made to the differential dataflow infrastructure that you have stopped changing this input at all times not greater or equal to this input time, and the system can now start to make progress determining the corresponding output changes. Cruciall, the calls to advance_to will be buffered until you call input.flush(), which exposes this information to the underlying timely dataflow system. This allows you to call advance_to as frequently as once per record, without overwhelming the underlying system.

when should i use advance_to? what happend to advance_to? can you show some sample code?
and input.flush? when and why should i use this? what happend ?

advance_to is only a task but not submitied? flush will submit all the task i submit? but if advance_to is a task submit,what is time? can i advance_to the same time many times? what will happen?

For example, our first example computations didn't call worker.step() explicitly, but just exited once it supplied the input changes. Exiting causes all of the work to happen (and complete, as the inputs are automatically closed as they are dropped). Explicit calls to worker.step() are important when we are maintaining interactive access to probes, and do not want to simply complete the computation.

so when do we call this worker.step?

there is a lot more confusion when i read the book...maybe english is not my native language cause some misunderstandings.

when i run the example showed in github mainpage,i think i know someting...but when i read the doc,i'm totally crazy...

thank you in advance~

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the chapter 1 and chapter 4 documentation referenced in the issue, then compare those explanations with the examples on the GitHub repository main page. Document what (data, time, diff) represents and clarify when advance_to, input.flush(), and worker.step() are used, including sample code. Done means the getting-started and chapter 4 material is understandable and aligned with the repository examples.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
documentation
Issue type
Documentation
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.