TimelyDataflow / TimelyDataflow/timely-dataflow
Dataflow construction seems slow-ish
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 3.6k
- Forks
- 293
- Avg merge
- 14h 46m
- Merged PRs (30d)
- 4
Description
I'm not certain this is a real issue, but the event_driven example can be made to create 1000 dataflows each a sequence of 1000 nodes, and doing so takes about 10s on my modern desktop. That seems like a long time to put together what is meant to be a linear amount of information. It does seem to be linear (scaling up either number by 10 changes the time by 10), but it would still be good to get a sense for what is going on. Initial investigation suggests allocations in reachability, which indeed does have a bunch of small per-operator allocations. Perhaps those could be consolidated somehow?
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Reproduce the reported case using the event_driven example: 1000 dataflows of 1000 nodes, and measure construction time. Then inspect the reachability path and its per-operator allocations using profiling or allocation measurements. Done means explaining the bottleneck and demonstrating a measured improvement or confirming that no actionable change is warranted.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- distributed-systems, performance
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100