TimelyDataflow / TimelyDataflow/differential-dataflow
All algorithms should be usable with generic containers
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 3k
- Forks
- 211
- Avg merge
- 10h 42m
- Merged PRs (30d)
- 34
Description
Differential offers many algorithms that absorb input data in some form. Most (all?) are compatible with vector-based data, and some support generic containers, or specific instances of containers other than vectors. We'd like to support generic containers or make it possible to easily integrate new specific containers for all parts of Differential. Here's an overview where we stand:
- arrange_core: Can absorb all data for which we have a merge batcher.
- Merge batcher: Generic over input/output data, but sometimes lacks flexibility, and is complex.
- The merge batcher should not name
Input, but be generic over some variant ofPush.
- The merge batcher should not name
- Half-join: Requires
Vec - Collection: Type can have generic containers, names owned
D/T/R, but not very usable for other containers thanVec.- Map only for
Vec -
map_in_place -
flat_map -
filter -
explode -
join_function -
enter/leave -
enter_at -
delay -
inspect -
inspect_batch -
assert_empty: needsconsolidate -
consolidate/consolidate_named: Work-around to build arrangement. -
consolidate_stream: UsesConsolidatingContainerBuilder -
negateis aware of containers (traitoperators::Negate), but generic parameters are in wrong order. See #587
- Map only for
-
ConsolidatingContainerBuilder - Remove
TimelyStack
There are a bunch of types that we don't use a lot, so the list is likely incomplete.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the unchecked items in the issue, especially the merge batcher, half-join, and Collection APIs, then trace how they currently depend on Vec and the listed container traits. Choose one narrowly scoped subsystem before investigating broader changes such as ConsolidatingContainerBuilder or removing TimelyStack. Done should mean the selected subsystem supports generic containers without regressing its existing behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- data-engineering, distributed-systems
- Issue type
- Refactor
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100