aws-samples / aws-samples/amazon-cloudwatch-container-insights

cannot get pod from kubelet, err: call to /pods endpoint failed:

Open
#160 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Shell
Stars
178
Forks
117
Avg merge
1d 10h
Merged PRs (30d)
3

Description

Seeing below Error when trying to deploy latest cloudwatch-agent (1.3xxx) in aws eks version 1.27 and 1.28 with IMDSv2 enabled.
Cloudwatch agent 1.2xx was working as expected with similar AWS EKS Configuration.

2024-03-14T17:43:45Z I! {"caller":"service@v0.89.0/service.go:143","msg":"Starting CWAgent...","Version":"1.300034.0b498","NumCPU":8}
2024-03-14T17:43:45Z I! {"caller":"extensions/extensions.go:34","msg":"Starting extensions..."}
2024-03-14T17:43:45Z I! {"caller":"extensions/extensions.go:37","msg":"Extension is starting...","kind":"extension","name":"agenthealth/logs"}
2024-03-14T17:43:45Z I! {"caller":"extensions/extensions.go:45","msg":"Extension started.","kind":"extension","name":"agenthealth/logs"}
2024-03-14T17:43:45Z I! {"caller":"host/ec2metadata.go:78","msg":"Fetch instance id and type from ec2 metadata","kind":"receiver","name":"awscontainerinsightreceiver","data_type":"metrics"}
2024-03-14T17:43:45Z I! {"caller":"service@v0.89.0/service.go:178","msg":"Starting shutdown..."}
2024-03-14T17:43:45Z I! {"caller":"extensions/extensions.go:52","msg":"Stopping extensions..."}
2024-03-14T17:43:45Z I! {"caller":"service@v0.89.0/service.go:192","msg":"Shutdown complete."}
Error: cannot start pipelines: cannot get pod from kubelet, err: call to /pods endpoint failed: Get "https://x.x.x.x:y/pods": Forbidden
2024-03-14T17:43:45Z E! [telegraf] Error running agent: cannot start pipelines: cannot get pod from kubelet, err: call to /pods endpoint failed: Get "https://x.x.x.x:yyy/pods": Forbidden

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the CloudWatch agent 1.300034.0b498 startup on AWS EKS 1.27 or 1.28 with IMDSv2 enabled, focusing on the awscontainerinsightreceiver request to the kubelet /pods endpoint. Compare the behavior with agent 1.2xx and inspect the reported Forbidden response; done means the agent starts its pipelines and retrieves pod data successfully.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, kubernetes
Domain
cloud, devops, observability-sre
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.