aws / aws/amazon-cloudwatch-agent
[k8s] Pod metrics is gone when using containerd as runtime
- Dominant language
- Go
- Stars
- 550
- Forks
- 271
- Avg merge
- 1d 21h
- Merged PRs (30d)
- 13
Description
This is exported from internal ticket
## TL;DR
The latest image is released, if you were using temp image from this comment https://github.com/aws/amazon-cloudwatch-agent/issues/188#issuecomment-803764697 please update to the latest tag.
If the error message `W! No pod metric collected, metrics count is still 7 is containerd socket mounted? https://github.com/aws/amazon-cloudwatch-agent/issues/188`leads you to this issue
- make sure you have updated the yaml to mount the containerd socket into the cloudwatch-agent pod
- the path for containerd socket may not be in the standard location. e.g. bottlerocket uses `/run/dockershim.sock` instead of `/run/containerd/containerd.sock`
## Background
We were relying on pause container to have `POD` for detecting pod, which is the case for docker but not for containerd https://github.com/containerd/cri/issues/922#issuecomment-423729537
User will not see pod metrics in container insight dashboard and they will find the following log which is introduced in #171
https://github.com/aws/amazon-cloudwatch-agent/blob/fbdd619269be7a00172e06992a8d40b22be1a6d7/plugins/inputs/cadvisor/container_info_processor.go#L72-L72
The root cause is we are expecting `containerName == 'POD'` to mark a path as pod
https://github.com/aws/amazon-cloudwatch-agent/blob/fbdd619269be7a00172e06992a8d40b22be1a6d7/plugins/inputs/cadvisor/container_info_processor.go#L119-L126
## Fix
- code should be similar to https://github.com/DataDog/integrations-core/pull/2283
- manifest need to mount containerd sock into cwagent daemonset container
## Release
The fix will be included in next release, the release date is not determined (yet).
Contributor guide
Assessment
This issue has not been assessed yet.