TimelyDataflow / TimelyDataflow/differential-dataflow

Use descriptions for compaction

Open
#37 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Rust
Stars
3k
Forks
211
Avg merge
10h 42m
Merged PRs (30d)
34

Description

Batches come with "descriptions", which contain lower and upper bounds on the timestamps of updates contained within. Specifically, the times within are those greater or equal to some element of lower and not greater or equal to any element of upper.

It seems like we should be able to use these bounds to inform advance compaction, in which we consider advancing all times in a batch by a provided frontier. The frontier's interaction with lower and upper seems like it constrains what happens to the times in the batch, and it could tell us about whether we should expect the frontier to collapse all times down, or collapse no times at all, or what.

This information could let us advance more aggressively when appropriate, and to put off compaction when we can see that the involved batch is unlikely to be much improved.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

The issue names no files, tests, or concrete entry points beyond advance compaction and batch descriptions containing lower and upper bounds. First locate the implementation of advance and the representation of these descriptions, then clarify the intended frontier behavior; done requires an agreed algorithm and validation for when compaction should be accelerated or deferred.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
data-engineering, distributed-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.