TimelyDataflow / TimelyDataflow/differential-dataflow

Consider LSM for `operators::ValueHistory`

Open
#82 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Rust
Stars
3k
Forks
211
Avg merge
10h 42m
Merged PRs (30d)
34

Description

The ValueHistory type is used to wrap the (V, T, R) triples describing the values associated with a key, and their history. Its intent is to provide efficient replay of the history, from which it can produce the corresponding value collection at any point along the way.

At the moment, this works by ordering edits by time and incorporating edits as we move forward in time. As time is not totally ordered, "incorporated" edits hang out, and require rescanning and reaccumulation to determine the actual state of the collection at any time.

It seems not unreasonable to think about incorporating these edits in a LSM (log structured merge) fashion, where we maintain multiple sorted lists of (V, T, R) triples that are "in effect", which we merge live as the user navigates around the values.

This has some advantages and some disadvantages, compared to the current approach which is to re-sort every time a batch of triples come "into effect".

  1. The main disadvantage is complexity: for small histories, just sorting the updates is pretty easy, especially if the consumer wants access to the whole collection of values anyhow.

  2. Some amount of time is spent reforming the current input collection, currently by collecting input differences less or equal to the current time, then sorting and coalescing updates. Much of this work could become redundant with a LSM structure that keeps the data sorted by value. We would need to scan the histories for each value, to accumulate the diffs, but the values would already be in-order and the accumulation doesn't require reorganizing any data.

  3. Consider even more interesting computations, where the consumer may only want a subset of the values, either a prefix (for "top k" computations) or random access to data-driven locations (for intersection, e.g. in GenericJoin). In these cases, what would have been linear work for each update becomes amortized logarithmic, and if the query only required some few elements, this could be a substantial reduction.

  4. Going further, it seems plausible that we could load data from the underlying collection traces on demand, only loading the history for a value when we actually need it (having ValueHistory act as a cache in front of the collection trace). This would reduce the fixed cost of ingesting the data for a key, and extend the benefits to cases where there may just be a few updates to apply to a large collection.

Almost all of the data are already organized as (V, T, R) triples sorted by V, even in the underlying collection trace, so it seems reasonable that we might be able to wrap this representation, and do relatively little work when pulling in triples from a collection trace. The only "reorganization" that I see is that we want to be able to present the times at which group may need to re-evaluate its logic, which requires the times of updates sorted by their total order (but which does not require reorganizing the data).

One concern is that we may not have good benchmarks for evaluating this. Computations like bfs perform group operations on relatively small groups, so we may need to investigate more skewed degree distributions to provoke the scary quadratic behavior. Alternately, queries like TPCH Q15 perform a maximum, currently implemented hierarchically, but if done with a flat implementation would immediately suffer from quadratic behavior. Q15 has the problem that we won't see any benefit until we do the more exotic "lazy" access to data.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

No files or tests are named. Start at operators::ValueHistory and trace its current time-ordered replay and rescanning behavior, then inspect how group and collection traces use it. Investigate benchmark coverage for bfs and TPCH Q15, especially skewed workloads; done would require evidence that an LSM or lazy-loading design improves the identified cases.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
data-engineering, performance
Issue type
Refactor
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.