TimelyDataflow / TimelyDataflow/differential-dataflow
Reduce operator (and others) may be quadratic with deep history
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 3k
- Forks
- 211
- Avg merge
- 10h 42m
- Merged PRs (30d)
- 34
Description
We have reports from the field that when provided with many fine grained updates, even to a collection that has a small working set. Looking at the code, it seems like there are possible moments where this could happen, for example we build a CursorList of available batches, but navigation through a cursor list will be quadratic with the number of batches linked up (they were intended for LSM shaped data). This feels like a suboptimal way to handle large numbers of batches, especially if it expected that they will soon be (or already have been) merged.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by tracing how operators build and navigate the CursorList of available batches, then profile workloads with many fine-grained updates and deep history. Done means the affected operators no longer show quadratic behavior as the number of linked batches grows, especially when batches are soon merged.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- data-engineering, performance
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100