TimelyDataflow / TimelyDataflow/differential-dataflow

Spec out Trace Adapters?

Open
#36 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Rust
Stars
3k
Forks
211
Avg merge
10h 42m
Merged PRs (30d)
34

Description

Traces represent a pile of updates (key, val, time, diff), indexed by key then val then time. There are several transformations on collections that we could also apply to these indexed representations. The most appealing example I can think of is enter, which brings a collection into a loop, changing its timestamp. We could imagine defining a graph in the outermost streaming scope, and then import the single indexed representation into multiple iterative contexts, sharing the state between all of them.

This requires a bit of thinking to understand how easily we can implement the Trace trait, and all of its dependent traits, for something as simple as "put a zero at the end of each timestamp". There are possibly some issues about types; for example, it is harder to return references to timestamp if none of those timestamps actually exist in the trace (the trace cursors will probably have to stage the timestamps in local variables, perhaps).

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by examining the Trace trait and its dependent traits, then compare them with the indexed representation of (key, val, time, diff) and the proposed enter transformation. A useful outcome would specify whether trace adapters can support staged timestamps and shared state across iterative contexts, including the relevant type constraints.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
data-engineering, distributed-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.