slow lc_lib:get_sys_memory_usage() with lots of ports
- Dominant language
- Erlang
- Stars
- 6
- Forks
- 4
- PR merge metrics
- No merged PRs in 30d
Description
## Description
We get report that during 100M connections tests, the function `lc_lib:get_sys_memory_usage()` is very slow as following.
```
iex(emqx@emqx-6.int.thales)9> :timer.tc(&:load_ctl.get_sys_memory/0)
{9624, {0.8642148373449763, 134799187968}}
iex(emqx@emqx-6.int.thales)10> :timer.tc(&:load_ctl.get_sys_memory/0)
{717, {0.8661882218141999, 134799187968}}
iex(emqx@emqx-6.int.thales)11> :timer.tc(&:load_ctl.get_sys_memory/0)
{609, {0.8671846375495222, 134799187968}}
iex(emqx@emqx-6.int.thales)12> :timer.tc(&:load_ctl.get_sys_memory/0)
{4690, {0.8890156894153435, 134799187968}}
iex(emqx@emqx-6.int.thales)13> :timer.tc(&:load_ctl.get_sys_memory/0)
{617, {0.890001986632739, 134799187968}}
iex(emqx@emqx-6.int.thales)14> :timer.tc(&:load_ctl.get_sys_memory/0)
{64898181, {0.8134260153112319, 134799187968}}
```
## Impacted nodes
10/20 nodes runs into this issue.
## Read from shell
Reading Cgroup/ cgroupv2/ or `/proc/meminfo` from shell is fast.
## runq length:
```
iex(emqx@emqx-6.int.thales)18> :erlang.statistics(:run_queue_lengths) |> IO.inspect(limit: :infinity)
[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
```
We suspect it relates to the large number of erlang ports. (~5M / node)
## cause
Likely it is caused by `memsup`
```
iex(emqx@emqx-6.int.thales)18> :timer.tc(&:memsup.get_system_memory_data/0)
{13702147,
[
available_memory: 86003736576,
buffered_memory: 119590912,
cached_memory: 4353019904,
free_memory: 81441886208,
free_swap: 0,
system_total_memory: 134799183872,
total_memory: 134799183872,
total_swap: 0
]}
```
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.