TimelyDataflow / TimelyDataflow/differential-dataflow
Insertion sort for consolidation
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 3k
- Forks
- 211
- Avg merge
- 10h 42m
- Merged PRs (30d)
- 34
Description
We very often "consolidate" collections of updates, triples (data, time, diff), so that there is at most one element with each data and time. This is currently done by sorting such a collection by (data, time) and then swinging through the result.
Most sorting algorithms use insertion sort on small sets, either in the recursive base case for quicksort or to form sorted runs for mergesort. We could perform a similar action when adding elements to a "compactable list", with the improvement that if we find a matching element it can be compacted in place. Ideally this would be no worse than sorting the data (although it probably does work that subsequent sorting can't exploit perfectly), and often much better (when adjacent elements collapse, or when the set of data and times is small).
More generally, we should be mindful that consolidation is not always cheap (we must allocate the backing memory, and then perform super-linear sorting), and a solid implementation of compaction should have pervasive performance wins.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
The issue names no files, tests, or entry points to begin from. Investigate the existing compactable-list and consolidation implementation, then establish benchmarks and tests showing insertion-time compaction is correct and improves consolidation performance before defining the implementation scope.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- data-engineering, performance
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100