TimelyDataflow / TimelyDataflow/differential-dataflow

Implement a GroupTotal trait and operators

Open
#76 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Rust
Stars
3k
Forks
211
Avg merge
10h 42m
Merged PRs (30d)
34

Description

Issue #74 calls for specialized implementations for totally ordered timestamps. The group and group_u operators are currently very complex, in part due to the complexity of partially ordered times. This is a non-trivial implementation (much more complex than CountTotal or DistinctTotal), but there are at least a few important simplifications we can exploit:

  1. The times at which the group variants must be re-evaluated are exactly those found in the input differences. This is unlike for partially ordered times, which have a more complicated logic to determine further synthetic times that may produce output differences.

  2. The running accumulations for each key need not (i) track the meet of subsequent times (it would be just the min, equal to the next time in sorted order), nor (ii) maintain an intermediate collection of updates that have been advanced up to this meet (it can instead maintain a simpler running accumulation of the collection in question).

There are still a large number of subtleties, and the current group.rs file may not be the best starting point due to its complexity (both intrinsic complexity, and questionable structure). I recommend if anyone wants to try and work on this issue we should get in touch and map out the right way to spec out how it should work.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with group.rs and trace the existing group and group_u operators, then compare the simpler CountTotal and DistinctTotal implementations. Work out the specification for GroupTotal with totally ordered timestamps, including reevaluation at input-difference times and running accumulations. Done means the specialized trait and operators are implemented after their subtleties and structure have been mapped out.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
stream-processing
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.