TimelyDataflow / TimelyDataflow/timely-dataflow

Changing default allocation size in `channel/mod.rs`

Open
#346 6 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Rust
Stars
3.6k
Forks
293
Avg merge
14h 46m
Merged PRs (30d)
4

Description

I am creating a large dataflow graph with ~10000 nodes. It works great, except that I noticed that the static memory footprint of the program (before I feed any data to it) keeps growing as the graph gets bigger. I think it is currently in the order of a 100MB. This does not sound too bad, but my application creates thousands of instances of the dataflow, at which point this overhead becomes significant, and in fact dominates the memory footprint of the program. It appears that the main contributor are the buffers of size Message::default() (currently equal to 1024), pre-allocated for all channels. Is there a way to change this default without forking the repo?

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in channel/mod.rs and inspect how Message::default() determines the pre-allocated channel buffer size. Trace where that default is used when channels are created, then check existing tests around channel allocation if present. Done means the allocation size can be changed without forking the repository and the behavior is covered by an appropriate test.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
distributed-systems
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.