TimelyDataflow / TimelyDataflow/differential-dataflow
Remove `distinguish_since` capability
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 3k
- Forks
- 211
- Avg merge
- 10h 42m
- Merged PRs (30d)
- 34
Description
Currently any TraceAgent, the thing you hold on to to be able to re-use an arrangement, has two capabilities (with horrible names):
-
advance_by: indicates the lower bound (frontier) on times that maybe be used as comparisons against the entries in the trace. As this advances, the trace is able to compact its own representation by determining that some times are equivalent, and their differences coalesced. -
distinguish_since: independently, each trace maintains boundaries between each of the batches it has created, and waits for trace agents to decline an interest in keeping them from merging. This is so that operators likejoincan get access to what may be historical traces, because theyadvance_bya trace based on the progress in the other input (weird, but true).
The times in the distinguish_since capability currently have no particular semantic use, and could as easily be replaced with the sequence number of the batch. Although one could imagine a general and rich "bookmarking" functionality, the current use is "for this batch, could I have the trace up to but not including it, please".
Rather than track two separate capabilities, which causes a fair amount of headache (especially when they head in different directions), I propose that we scrap the distinguish_since capability, and simply capture the state of the trace at the moment each batch is created (in the form of a list of batches). The associated streams of data would change from "batch" to "batch + history", and allow each implementation to hold on to the history as long as appropriate, without blocking other users of the trace.
One possible downside of this is that the distinguish_since capability was used as a way of blocking merging until a user had downgraded their advance_by capabilities, increasing the chance that when time came for a merge there would be the potential to collapse the updates as well. Now as soon as a batch is entered into the trace it can be merged, even if it would be best overall for that merging to be delayed just a moment. Perhaps we can build in a one-call delay to give each downstream operator a chance to perform the associated work.
This might also be a good time to rename the advance_by capability, which describes what the trace does to each of the times (they are "advanced by" the frontier), but isn't especially descriptive for the user of the trace. Awkwardly, it seems that "distinguish since" is a lot closer to what the capability represents (all times in advance of the frontier must be distinguishable from each other). I'd prefer not to simply swap that name in, in the interest of shaking out bugs where we forget to rename things, but it might be a good time to float some synonyms.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by locating TraceAgent and the implementations of the distinguish_since and advance_by capabilities. Read how trace batches and their associated streams are represented, then map the downstream join behavior described in the issue. Done means the separate capability is removed and batch history remains available without it, with the affected behavior validated.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- data-engineering, distributed-systems
- Issue type
- Refactor
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100