TimelyDataflow / TimelyDataflow/timely-dataflow

Too many send and receive thread in timely processor

Open
#153 4 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Rust
Stars
3.6k
Forks
293
Avg merge
14h 46m
Merged PRs (30d)
4

Description

I use a few hundreds processors in my job, and I found that each processor has too many send and receive thread as follows.
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
16969 admin 20 0 6855696 2.694g 46596 S 0.0 0.7 0:00.11 send thread 0
16970 admin 20 0 6855696 2.694g 46596 S 0.0 0.7 0:00.04 recv thread 0
16971 admin 20 0 6855696 2.694g 46596 S 0.0 0.7 0:00.12 send thread 1
16972 admin 20 0 6855696 2.694g 46596 S 0.0 0.7 0:00.05 recv thread 1
16973 admin 20 0 6855696 2.694g 46596 S 0.0 0.7 0:00.12 send thread 2
16974 admin 20 0 6855696 2.694g 46596 S 0.0 0.7 0:00.07 recv thread 2
16975 admin 20 0 6855696 2.694g 46596 S 0.0 0.7 0:00.12 send thread 3
16976 admin 20 0 6855696 2.694g 46596 S 0.0 0.7 0:00.04 recv thread 3
16977 admin 20 0 6855696 2.694g 46596 S 0.0 0.7 0:00.09 send thread 4
16978 admin 20 0 6855696 2.694g 46596 S 0.0 0.7 0:00.01 recv thread 4
16979 admin 20 0 6855696 2.694g 46596 S 0.0 0.7 0:00.12 send thread 5
16980 admin 20 0 6855696 2.694g 46596 S 0.0 0.7 0:00.06 recv thread 5
16981 admin 20 0 6855696 2.694g 46596 S 0.0 0.7 0:00.12 send thread 6
16982 admin 20 0 6855696 2.694g 46596 S 0.0 0.7 0:00.03 recv thread 6
16983 admin 20 0 6855696 2.694g 46596 S 0.0 0.7 0:00.11 send thread 7
16984 admin 20 0 6855696 2.694g 46596 S 0.0 0.7 0:00.01 recv thread 7
16985 admin 20 0 6855696 2.694g 46596 S 0.0 0.7 0:00.12 send thread 8
16986 admin 20 0 6855696 2.694g 46596 S 0.0 0.7 0:00.07 recv thread 8
16987 admin 20 0 6855696 2.694g 46596 S 0.0 0.7 0:00.11 send thread 9
16988 admin 20 0 6855696 2.694g 46596 S 0.0 0.7 0:00.00 recv thread 9
16989 admin 20 0 6855696 2.694g 46596 S 0.0 0.7 0:00.11 send thread 10
16990 admin 20 0 6855696 2.694g 46596 S 0.0 0.7 0:00.05 recv thread 10
16991 admin 20 0 6855696 2.694g 46596 S 0.0 0.7 0:00.10 send thread 11
16992 admin 20 0 6855696 2.694g 46596 S 0.0 0.7 0:00.04 recv thread 11

When I use 400 processors in my job, there will be 800 send/recv threads in each processor. I think it should be optimized for there are too many threads in one processor. What do you think? Thanks @frankmcsherry

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

The report names no source files, tests, or entry points; begin by reproducing the thread count with a job using 400 processors and tracing where timely processors create send and receive threads. Done means deciding whether the per-processor thread count should change and validating the chosen behavior at that scale.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
distributed-systems, performance
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.