deepseek-ai / deepseek-ai/DeepEP

[Performance] Warn if automatic NUMA balancing of the Linux kernel is enabled

Open
#624 2 comments 5 reactions 0 assignees View on GitHub
Dominant language
Cuda
Stars
10.1k
Forks
1.4k
Avg merge
2d 17h
Merged PRs (30d)
3

Description

The performance of `task_numa_work` is unacceptable on Linux 5.15.
```txt
# funclatency -d 60 -m -p 2256971 task_numa_work
Tracing 1 functions for "task_numa_work"... Hit Ctrl-C to end.

msecs : count distribution
0 -> 1 : 72 |****************************************|
2 -> 3 : 8 |**** |
4 -> 7 : 0 | |
8 -> 15 : 57 |******************************* |
16 -> 31 : 2 |* |

avg = 5 msecs, total: 789 msecs, count: 139

Detaching...
```

It may add 16+ ms of latency to methods such as `deep_ep::Buffer::internode_dispatch`.
```
sglang::schedul 2256971 53718.148983: 740539 cycles:
ffffffffba32fa60 vm_normal_page+0x0 ([kernel.kallsyms])
ffffffffba342027 change_pte_range+0x1a7 ([kernel.kallsyms])
ffffffffba34266d change_pmd_range.isra.0+0x15d ([kernel.kallsyms])
ffffffffba342a3e change_protection_range+0x16e ([kernel.kallsyms])
ffffffffba37d536 change_prot_numa+0x36 ([kernel.kallsyms])
ffffffffba14094b task_numa_work+0x1ab ([kernel.kallsyms])
ffffffffba11fc0c task_work_run+0x5c ([kernel.kallsyms])
ffffffffba1ad21d exit_to_user_mode_loop+0x11d ([kernel.kallsyms])
ffffffffba1ad28e exit_to_user_mode_prepare+0x6e ([kernel.kallsyms])
ffffffffbabb7385 irqentry_exit_to_user_mode+0x5 ([kernel.kallsyms])
ffffffffbac00d42 asm_sysvec_apic_timer_interrupt+0x12 ([kernel.kallsyms])
7ffdd57eac20 __vdso_clock_gettime+0x0 ([vdso])
7f17d22e566b clock_gettime@@GLIBC_2.17+0x1b (/usr/lib/x86_64-linux-gnu/libc.so.6)
7f17cf4a610d std::chrono::_V2::system_clock::now+0x1d (/usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.30)
7f10d4235330 deep_ep::Buffer::internode_dispatch+0x1f70 (/usr/local/lib/python3.12/dist-packages/deep_ep_cpp.cpython-312-x86_64-linux-gnu.so)
7f10d425fda9 pybind11::cpp_function::initialize, std::optional, std::optional, std::vector
7f10d4252eb0 pybind11::cpp_function::dispatcher+0x8a0 (/usr/local/lib/python3.12/dist-packages/deep_ep_cpp.cpython-312-x86_64-linux-gnu.so)
56d11a [unknown] (/usr/bin/python3.12)
53f5d9 _PyObject_MakeTpCall+0x2f9 (/usr/bin/python3.12)
54a2dc _PyEval_EvalFrameDefault+0x6fc (/usr/bin/python3.12)
5999ba [unknown] (/usr/bin/python3.12)
54f527 _PyEval_EvalFrameDefault+0x5947 (/usr/bin/python3.12)
```

Contributor guide

No contributing guide indexed for this repository

Research direction

No repository file or test is named. Start by tracing deep_ep::Buffer::internode_dispatch and the surrounding runtime or initialization path, then verify how the Linux task_numa_work latency is detected; done means users receive a warning when automatic NUMA balancing is enabled.

Written by the indexing model from the issue text.

Assessment

Tech stack
linux
Domain
operating-systems, performance
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.