emqx / emqx/lc

slow lc_lib:get_sys_memory_usage() with lots of ports

Open
#13 0 comments 1 reaction 0 assignees View on GitHub
Dominant language
Erlang
Stars
6
Forks
4
PR merge metrics
No merged PRs in 30d

Description

## Description
We get report that during 100M connections tests, the function `lc_lib:get_sys_memory_usage()` is very slow as following.

```
iex(emqx@emqx-6.int.thales)9> :timer.tc(&:load_ctl.get_sys_memory/0)
{9624, {0.8642148373449763, 134799187968}}
iex(emqx@emqx-6.int.thales)10> :timer.tc(&:load_ctl.get_sys_memory/0)
{717, {0.8661882218141999, 134799187968}}
iex(emqx@emqx-6.int.thales)11> :timer.tc(&:load_ctl.get_sys_memory/0)
{609, {0.8671846375495222, 134799187968}}
iex(emqx@emqx-6.int.thales)12> :timer.tc(&:load_ctl.get_sys_memory/0)
{4690, {0.8890156894153435, 134799187968}}
iex(emqx@emqx-6.int.thales)13> :timer.tc(&:load_ctl.get_sys_memory/0)
{617, {0.890001986632739, 134799187968}}
iex(emqx@emqx-6.int.thales)14> :timer.tc(&:load_ctl.get_sys_memory/0)
{64898181, {0.8134260153112319, 134799187968}}
```

## Impacted nodes

10/20 nodes runs into this issue.

## Read from shell
Reading Cgroup/ cgroupv2/ or `/proc/meminfo` from shell is fast.

## runq length:
```
iex(emqx@emqx-6.int.thales)18> :erlang.statistics(:run_queue_lengths) |> IO.inspect(limit: :infinity)
[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
```

We suspect it relates to the large number of erlang ports. (~5M / node)

## cause

Likely it is caused by `memsup`
```
iex(emqx@emqx-6.int.thales)18> :timer.tc(&:memsup.get_system_memory_data/0)
{13702147,
[
available_memory: 86003736576,
buffered_memory: 119590912,
cached_memory: 4353019904,
free_memory: 81441886208,
free_swap: 0,
system_total_memory: 134799183872,
total_memory: 134799183872,
total_swap: 0
]}
```

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.