TimelyDataflow / TimelyDataflow/differential-dataflow

All algorithms should be usable with generic containers

Open
#622 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Rust
Stars
3k
Forks
211
Avg merge
10h 42m
Merged PRs (30d)
34

Description

Differential offers many algorithms that absorb input data in some form. Most (all?) are compatible with vector-based data, and some support generic containers, or specific instances of containers other than vectors. We'd like to support generic containers or make it possible to easily integrate new specific containers for all parts of Differential. Here's an overview where we stand:

  • arrange_core: Can absorb all data for which we have a merge batcher.
  • Merge batcher: Generic over input/output data, but sometimes lacks flexibility, and is complex.
    • The merge batcher should not name Input, but be generic over some variant of Push.
  • Half-join: Requires Vec
  • Collection: Type can have generic containers, names owned D/T/R, but not very usable for other containers than Vec.
    • Map only for Vec
    • map_in_place
    • flat_map
    • filter
    • explode
    • join_function
    • enter/leave
    • enter_at
    • delay
    • inspect
    • inspect_batch
    • assert_empty: needs consolidate
    • consolidate/consolidate_named: Work-around to build arrangement.
    • consolidate_stream: Uses ConsolidatingContainerBuilder
    • negate is aware of containers (trait operators::Negate), but generic parameters are in wrong order. See #587
  • ConsolidatingContainerBuilder
  • Remove TimelyStack

There are a bunch of types that we don't use a lot, so the list is likely incomplete.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the unchecked items in the issue, especially the merge batcher, half-join, and Collection APIs, then trace how they currently depend on Vec and the listed container traits. Choose one narrowly scoped subsystem before investigating broader changes such as ConsolidatingContainerBuilder or removing TimelyStack. Done should mean the selected subsystem supports generic containers without regressing its existing behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
data-engineering, distributed-systems
Issue type
Refactor
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.