TimelyDataflow / TimelyDataflow/differential-dataflow
Revisit the stashing logic in MergeBatcherColumnation
@antiguru is already working on this.
Since Nov 23, 2023.
- Dominant language
- Rust
- Stars
- 3k
- Forks
- 211
- Avg merge
- 10h 42m
- Merged PRs (30d)
- 34
Description
As surfaced recently, the stashing logic to recycle empty buffers in the merge batcher is subtle; it should make sure there are at most 2 empty buffers, which is all it will ever need. The vector-based merge batcher implements this, but the logic is not simple to understand. When implementing the columnation-based merge batcher, I got it wrong, causing the seal function to temporarily retain all empty buffers, which caused an OOM situation.
We should revisit that the implementation maints the invariant that there are at most two empty buffers, and reason about why this is sufficient.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.