TimelyDataflow / TimelyDataflow/differential-dataflow
Small batches merge awkwardly
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 3k
- Forks
- 211
- Avg merge
- 10h 42m
- Merged PRs (30d)
- 34
Description
Because of timely dataflow's restrictions on single capabilities per message, we often end up with several very small batches and one large batch. This can lead to several merges between small and large batches, which is a fairly inefficient way to do things.
Relatively little is known about this, except that we spend a fair bit of time in merging, mostly allocating and dropping random bits of memory. If this is because of lots of skewed merges, we could look into scheduling merges in a different order that works to merge small batches together.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
The issue names no files, tests, or entry points. Begin by profiling merge activity and allocation time to determine whether skewed batch sizes are responsible, then investigate whether a different merge order reduces inefficient small-to-large merges.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- performance
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100