TimelyDataflow / TimelyDataflow/timely-dataflow

Dataflow construction seems slow-ish

Open
#308 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Rust
Stars
3.6k
Forks
293
Avg merge
14h 46m
Merged PRs (30d)
4

Description

I'm not certain this is a real issue, but the event_driven example can be made to create 1000 dataflows each a sequence of 1000 nodes, and doing so takes about 10s on my modern desktop. That seems like a long time to put together what is meant to be a linear amount of information. It does seem to be linear (scaling up either number by 10 changes the time by 10), but it would still be good to get a sense for what is going on. Initial investigation suggests allocations in reachability, which indeed does have a bunch of small per-operator allocations. Perhaps those could be consolidated somehow?

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Reproduce the reported case using the event_driven example: 1000 dataflows of 1000 nodes, and measure construction time. Then inspect the reachability path and its per-operator allocations using profiling or allocation measurements. Done means explaining the bottleneck and demonstrating a measured improvement or confirming that no actionable change is warranted.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
distributed-systems, performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.