elastic / elastic/apm-agent-python

APM agent fails platform detection on cgroupv2 based systems

Open
#1,691 3 comments 0 reactions 0 assignees View on GitHub
8.8-candidate agent-python community triage
Dominant language
Python
Stars
431
Forks
239
Avg merge
5d 10h
Merged PRs (30d)
7

Description

Hello!

Widespread cgroupv2 adoption is around the corner, as many popular distributions already come with cgroupv2 enabled by default.

The problem currently is with the fact, that the APM agent cannot detect that it is running on cgroupv2 based system, as it tries to parse the cgroup file which is fairly empty on a cgroupv2 enabled system. This causes the agent to specify "linux" as it's platform, omit container/pod ID's and cause multiple integrations to fail inside the Elastic ecosystem.

The current cgroup implementation uses the `/proc/self/cgroup` file to obtain relevant container and pod IDs. This file is empty on cgroupv2 based systems, and even in so called cgroupv1 systems, it's an undocumented "feature".

Currently there is no agreed upon method for obtaining this information from inside the container, and it's still a standing issue for the open container spec developers: https://github.com/opencontainers/runtime-spec/issues/1105

My proposal is to use the workaround similar to here:
- https://community.toradex.com/t/python-nullresource-error-when-running-torizoncore-builder-build/15240/4
- https://github.com/open-telemetry/opentelemetry-js-contrib/pull/1181/files

The `/proc/self/mountinfo` file still contains references to necessary information (pod and container uid).

We could use this until at least the standard is agreed upon, and could be switched out.

An example of the relevant information from my latest stable Flatcar Linux system, with Containerd, Kubernetes 1.24.7 and cgroupv2 enabled:

```
5051 5044 259:8 /lib/kubelet/pods/68ee930a-2bd8-447b-8deb-426add7a2d09/etc-hosts /etc/hosts rw,relatime - xfs /dev/nvme0n1p3 rw,seclabel,attr2,inode64,logbufs=8,logbsize=32k,noquota
5052 5046 259:8 /lib/kubelet/pods/68ee930a-2bd8-447b-8deb-426add7a2d09/containers//2631d4a6 /dev/termination-log rw,relatime - xfs /dev/nvme0n1p3 rw,seclabel,attr2,inode64,logbufs=8,logbsize=32k,noquota
5053 5044 259:8 /lib/containerd/io.containerd.grpc.v1.cri/sandboxes/199dafcfc5712cd1e9e49e94642e7df6cdf63356bbc3601e9115f26fd0d096e1/hostname /etc/hostname rw,relatime - xfs /dev/nvme0n1p3 rw,seclabel,attr2,inode64,logbufs=8,logbsize=32k,noquota
5054 5044 259:8 /lib/containerd/io.containerd.grpc.v1.cri/sandboxes/199dafcfc5712cd1e9e49e94642e7df6cdf63356bbc3601e9115f26fd0d096e1/resolv.conf /etc/resolv.conf rw,relatime - xfs /dev/nvme0n1p3 rw,seclabel,attr2,inode64,logbufs=8,logbsize=32k,noquota
```

The problem which prevents me from working on this issue is that I do not know which formats these lines can take on different systems.

What are your ideas, would this method work?

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.