aws-samples / aws-samples/amazon-cloudwatch-container-insights
Pod logs not getting pushed with FluentD with various errors
- Dominant language
- Shell
- Stars
- 178
- Forks
- 117
- Avg merge
- 1d 10h
- Merged PRs (30d)
- 3
Description
I have several jobs running and the logs which were earlier getting pushed to cloudwatch insights. I was able to access them by job name.
However, recently, I only see cloudwatch and fluentd pod logs without actual logs that I need for my jobs.
In some cases, I get this error on fluentd:
```unable to get http response from http://169.254.170.2/v2/metadata, error: unable to get response from http://169.254.170.2/v2/metadata, error: Get \"http://169.254.170.2/v2/metadata\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)\n```
And in others I get this on cloudwatch agent:
```
access ECS task metadata fail with response unable to get response from http://169.254.170.2/v2/metadata, error: Get \"http://169.254.170.2/v2/metadata\": context deadline exceeded (Client.Timeout exceeded while awaiting headers), assuming I'm not running in ECS.\nNo csm configuration found.\nNo metric configuration found.\nConfiguration validation first phase succeeded\
```
I am new to this and I do not fully understand how things work. Can someone please help me out in understanding what must be going wrong? Is this a version upgrade issue or something?
Thanks!
Contributor guide
Research direction
Start with the Fluentd and CloudWatch agent pod logs shown in the report, then investigate access to the ECS task metadata endpoint for the affected jobs. Compare the current logging setup with the previously working configuration; done means identifying the cause and restoring the jobs' logs in CloudWatch.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, kubernetes
- Domain
- cloud, observability-sre
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 20/100