TimelyDataflow / TimelyDataflow/differential-dataflow
Use descriptions for compaction
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 3k
- Forks
- 211
- Avg merge
- 10h 42m
- Merged PRs (30d)
- 34
Description
Batches come with "descriptions", which contain lower and upper bounds on the timestamps of updates contained within. Specifically, the times within are those greater or equal to some element of lower and not greater or equal to any element of upper.
It seems like we should be able to use these bounds to inform advance compaction, in which we consider advancing all times in a batch by a provided frontier. The frontier's interaction with lower and upper seems like it constrains what happens to the times in the batch, and it could tell us about whether we should expect the frontier to collapse all times down, or collapse no times at all, or what.
This information could let us advance more aggressively when appropriate, and to put off compaction when we can see that the involved batch is unlikely to be much improved.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
The issue names no files, tests, or concrete entry points beyond advance compaction and batch descriptions containing lower and upper bounds. First locate the implementation of advance and the representation of these descriptions, then clarify the intended frontier behavior; done requires an agreed algorithm and validation for when compaction should be accelerated or deferred.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- data-engineering, distributed-systems
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100