TimelyDataflow / TimelyDataflow/differential-dataflow

Implement `distinct` for arrangements by key and value.

Open
#208 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Rust
Stars
3k
Forks
211
Avg merge
10h 42m
Merged PRs (30d)
34

Description

At the moment distinct operates on arrangements only by key, with an empty value. It seems like we should be able to implement distinct on arrangements by key and value, where the distribution is by key but there are attendant values each of which we want to occur at most once. The existing reduce code is not sufficient, but the permitted cursor navigation seems appropriate to drive another implementation to the history for pairs (key, val) just as if they were the key themselves.

The potential benefit from this optimization would be more available sharing for Datalog style computations which often use distinct, and then join their results in subsequent rounds of derivation. They relatively rarely use a by-self arrangement for the join, and instead have some key in mind that we could also use for the distinct.

cc: @ryzhyk

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by locating the existing distinct implementation for arrangements and the reduce code it currently relies on, then review the permitted cursor navigation APIs. Implement and validate history for (key, value) pairs while retaining distribution by key, with each value occurring at most once; the issue provides no specific file or test entry point.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
distributed-systems
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.