TimelyDataflow / TimelyDataflow/differential-dataflow

Present lower-level operator interfaces

Open
#326 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Rust
Stars
3k
Forks
211
Avg merge
10h 42m
Merged PRs (30d)
34

Description

Several operators are written with interfaces that are meant to be "foolproof", in that you should not be able to mis-use them. Specifically, the join class of methods allow you to provide closures that act on data, but are not provided time or diff, as it is possible to mis-use that responsibility. At the same time, there are useful operators that can benefit from that information: specifically, there are time and diff manipulating linear operators (e.g. temporal filters, negate, etc) that you might want to fuse on to the join output.

We could present "unsafe" versions of several operators that expose these details, for power users who are confident that they can write not-incorrect logic, or at least who are willing to absorb that responsibility. The upside for these users is the ability to avoid producing outputs in cases where filtering would apply (e.g. temporal filters) or actions that move information from data to diff (e.g. explode style operators).

Examples that seem like they are clear candidates are the join variants, as well as dogsdogsdogs's lookup_map and half_join operators. It wouldn't be unreasonable to do reduce as well, though I don't have an immediate use case for it that would be easy to validate ("also fusing operators").

One straw man example would be an appropriately named join_idk_internal_unsafe that exposes the most generality we can manage, without concern about shepherding the user to correct behavior, and then target the existing join_ variants onto it, mapping their safer closures to the framework of the unsafe operator.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reading the existing join variants and the lookup_map, half_join, and possibly reduce operators to understand which interfaces hide time and diff. Compare the proposed unsafe interfaces with the safer variants, and treat the work as complete only when the scope and generality of the exposed operators are settled and validated by appropriate operator behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
data-engineering
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.