aws / aws/amazon-cloudwatch-agent

[k8s] Pod metrics is gone when using containerd as runtime

Open
#188 11 comments 0 reactions 1 assignee Claimed by @pingleig View on GitHub
area/k8s bug component/container-insight
Dominant language
Go
Stars
550
Forks
271
Avg merge
1d 21h
Merged PRs (30d)
13

Description

This is exported from internal ticket

## TL;DR

The latest image is released, if you were using temp image from this comment https://github.com/aws/amazon-cloudwatch-agent/issues/188#issuecomment-803764697 please update to the latest tag.

If the error message `W! No pod metric collected, metrics count is still 7 is containerd socket mounted? https://github.com/aws/amazon-cloudwatch-agent/issues/188`leads you to this issue

- make sure you have updated the yaml to mount the containerd socket into the cloudwatch-agent pod
- the path for containerd socket may not be in the standard location. e.g. bottlerocket uses `/run/dockershim.sock` instead of `/run/containerd/containerd.sock`

## Background

We were relying on pause container to have `POD` for detecting pod, which is the case for docker but not for containerd https://github.com/containerd/cri/issues/922#issuecomment-423729537

User will not see pod metrics in container insight dashboard and they will find the following log which is introduced in #171

https://github.com/aws/amazon-cloudwatch-agent/blob/fbdd619269be7a00172e06992a8d40b22be1a6d7/plugins/inputs/cadvisor/container_info_processor.go#L72-L72

The root cause is we are expecting `containerName == 'POD'` to mark a path as pod

https://github.com/aws/amazon-cloudwatch-agent/blob/fbdd619269be7a00172e06992a8d40b22be1a6d7/plugins/inputs/cadvisor/container_info_processor.go#L119-L126

## Fix

- code should be similar to https://github.com/DataDog/integrations-core/pull/2283
- manifest need to mount containerd sock into cwagent daemonset container

## Release

The fix will be included in next release, the release date is not determined (yet).

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.