TimelyDataflow / TimelyDataflow/differential-dataflow

Insertion sort for consolidation

Open
#49 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Rust
Stars
3k
Forks
211
Avg merge
10h 42m
Merged PRs (30d)
34

Description

We very often "consolidate" collections of updates, triples (data, time, diff), so that there is at most one element with each data and time. This is currently done by sorting such a collection by (data, time) and then swinging through the result.

Most sorting algorithms use insertion sort on small sets, either in the recursive base case for quicksort or to form sorted runs for mergesort. We could perform a similar action when adding elements to a "compactable list", with the improvement that if we find a matching element it can be compacted in place. Ideally this would be no worse than sorting the data (although it probably does work that subsequent sorting can't exploit perfectly), and often much better (when adjacent elements collapse, or when the set of data and times is small).

More generally, we should be mindful that consolidation is not always cheap (we must allocate the backing memory, and then perform super-linear sorting), and a solid implementation of compaction should have pervasive performance wins.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

The issue names no files, tests, or entry points to begin from. Investigate the existing compactable-list and consolidation implementation, then establish benchmarks and tests showing insertion-time compaction is correct and improves consolidation performance before defining the implementation scope.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
data-engineering, performance
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.