apple / apple/foundationdb

Provide cgroup aware machine metrics

Open
#4,138 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
C++
Stars
16.7k
Forks
1.6k
Avg merge
1d 20h
Merged PRs (30d)
126

Description

Currently FDB collects some machine metrics like memory which work fine for normal setups. Once you start to run FDB inside containers (or `cgroups`) you will get wrong/confusing machine metrics since some of the metrics are not `cgroup` aware like `/proc/meminfo` https://github.com/apple/foundationdb/blob/master/flow/Platform.actor.cpp#L454 this will provide the host machine metrics and not the cgroup metrics. An example output of FDB running in Kubernetes (with a memory limit of 8GiB):

```bash
Process performance details:
1.1.1.1:4500:tls ( 2% cpu; 1% machine; 0.001 Gbps; 0% disk IO; 0.4 GB / 98.9 GB RAM )
1.1.1.1:4502:tls ( 1% cpu; 1% machine; 0.001 Gbps; 0% disk IO; 0.4 GB / 98.9 GB RAM )
1.1.1.2:4500:tls ( 1% cpu; 9% machine; 0.001 Gbps; 0% disk IO; 0.4 GB / 133.3 GB RAM )
1.1.1.2:4502:tls ( 2% cpu; 9% machine; 0.001 Gbps; 0% disk IO; 0.4 GB / 133.3 GB RAM )
1.1.1.3:4500:tls ( 2% cpu; 11% machine; 0.001 Gbps; 0% disk IO; 0.4 GB / 92.4 GB RAM )
1.1.1.3:4502:tls ( 1% cpu; 11% machine; 0.001 Gbps; 0% disk IO; 0.4 GB / 92.4 GB RAM )
```

One approach would be to check if `/sys/fs/cgroup/memory/memory.stat` exists and if so use the metrics contained there (we would need to adjust the parsing).

Contributor guide

Open the contributing guide

Research direction

Start in flow/Platform.actor.cpp around line 454 and inspect how machine memory metrics are read from /proc/meminfo. Compare that path with the cgroup memory metrics under /sys/fs/cgroup/memory/memory.stat, including the required parsing changes. Done means containerized FDB reports cgroup-aware machine metrics instead of host memory values.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp, linux
Domain
observability, operating-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.