aws-samples / aws-samples/amazon-cloudwatch-container-insights
cannot get pod from kubelet, err: call to /pods endpoint failed:
- Dominant language
- Shell
- Stars
- 178
- Forks
- 117
- Avg merge
- 1d 10h
- Merged PRs (30d)
- 3
Description
Seeing below Error when trying to deploy latest cloudwatch-agent (1.3xxx) in aws eks version 1.27 and 1.28 with IMDSv2 enabled.
Cloudwatch agent 1.2xx was working as expected with similar AWS EKS Configuration.
2024-03-14T17:43:45Z I! {"caller":"service@v0.89.0/service.go:143","msg":"Starting CWAgent...","Version":"1.300034.0b498","NumCPU":8}
2024-03-14T17:43:45Z I! {"caller":"extensions/extensions.go:34","msg":"Starting extensions..."}
2024-03-14T17:43:45Z I! {"caller":"extensions/extensions.go:37","msg":"Extension is starting...","kind":"extension","name":"agenthealth/logs"}
2024-03-14T17:43:45Z I! {"caller":"extensions/extensions.go:45","msg":"Extension started.","kind":"extension","name":"agenthealth/logs"}
2024-03-14T17:43:45Z I! {"caller":"host/ec2metadata.go:78","msg":"Fetch instance id and type from ec2 metadata","kind":"receiver","name":"awscontainerinsightreceiver","data_type":"metrics"}
2024-03-14T17:43:45Z I! {"caller":"service@v0.89.0/service.go:178","msg":"Starting shutdown..."}
2024-03-14T17:43:45Z I! {"caller":"extensions/extensions.go:52","msg":"Stopping extensions..."}
2024-03-14T17:43:45Z I! {"caller":"service@v0.89.0/service.go:192","msg":"Shutdown complete."}
Error: cannot start pipelines: cannot get pod from kubelet, err: call to /pods endpoint failed: Get "https://x.x.x.x:y/pods": Forbidden
2024-03-14T17:43:45Z E! [telegraf] Error running agent: cannot start pipelines: cannot get pod from kubelet, err: call to /pods endpoint failed: Get "https://x.x.x.x:yyy/pods": Forbidden
Contributor guide
Research direction
Start by reproducing the CloudWatch agent 1.300034.0b498 startup on AWS EKS 1.27 or 1.28 with IMDSv2 enabled, focusing on the awscontainerinsightreceiver request to the kubelet /pods endpoint. Compare the behavior with agent 1.2xx and inspect the reported Forbidden response; done means the agent starts its pipelines and retrieves pod data successfully.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, kubernetes
- Domain
- cloud, devops, observability-sre
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100