TimelyDataflow / TimelyDataflow/differential-dataflow
Spec out Trace Adapters?
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 3k
- Forks
- 211
- Avg merge
- 10h 42m
- Merged PRs (30d)
- 34
Description
Traces represent a pile of updates (key, val, time, diff), indexed by key then val then time. There are several transformations on collections that we could also apply to these indexed representations. The most appealing example I can think of is enter, which brings a collection into a loop, changing its timestamp. We could imagine defining a graph in the outermost streaming scope, and then import the single indexed representation into multiple iterative contexts, sharing the state between all of them.
This requires a bit of thinking to understand how easily we can implement the Trace trait, and all of its dependent traits, for something as simple as "put a zero at the end of each timestamp". There are possibly some issues about types; for example, it is harder to return references to timestamp if none of those timestamps actually exist in the trace (the trace cursors will probably have to stage the timestamps in local variables, perhaps).
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by examining the Trace trait and its dependent traits, then compare them with the indexed representation of (key, val, time, diff) and the proposed enter transformation. A useful outcome would specify whether trace adapters can support staged timestamps and shared state across iterative contexts, including the relevant type constraints.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- data-engineering, distributed-systems
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100