TimelyDataflow / TimelyDataflow/differential-dataflow
Implement `distinct` for arrangements by key and value.
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 3k
- Forks
- 211
- Avg merge
- 10h 42m
- Merged PRs (30d)
- 34
Description
At the moment distinct operates on arrangements only by key, with an empty value. It seems like we should be able to implement distinct on arrangements by key and value, where the distribution is by key but there are attendant values each of which we want to occur at most once. The existing reduce code is not sufficient, but the permitted cursor navigation seems appropriate to drive another implementation to the history for pairs (key, val) just as if they were the key themselves.
The potential benefit from this optimization would be more available sharing for Datalog style computations which often use distinct, and then join their results in subsequent rounds of derivation. They relatively rarely use a by-self arrangement for the join, and instead have some key in mind that we could also use for the distinct.
cc: @ryzhyk
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by locating the existing distinct implementation for arrangements and the reduce code it currently relies on, then review the permitted cursor navigation APIs. Implement and validate history for (key, value) pairs while retaining distribution by key, with each value occurring at most once; the issue provides no specific file or test entry point.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- distributed-systems
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 42/100