aws-samples / aws-samples/amazon-cloudwatch-container-insights

Pod logs not getting pushed with FluentD with various errors

Open
#129 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Shell
Stars
178
Forks
117
Avg merge
1d 10h
Merged PRs (30d)
3

Description

I have several jobs running and the logs which were earlier getting pushed to cloudwatch insights. I was able to access them by job name.
However, recently, I only see cloudwatch and fluentd pod logs without actual logs that I need for my jobs.

In some cases, I get this error on fluentd:

```unable to get http response from http://169.254.170.2/v2/metadata, error: unable to get response from http://169.254.170.2/v2/metadata, error: Get \"http://169.254.170.2/v2/metadata\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)\n```

And in others I get this on cloudwatch agent:

```
access ECS task metadata fail with response unable to get response from http://169.254.170.2/v2/metadata, error: Get \"http://169.254.170.2/v2/metadata\": context deadline exceeded (Client.Timeout exceeded while awaiting headers), assuming I'm not running in ECS.\nNo csm configuration found.\nNo metric configuration found.\nConfiguration validation first phase succeeded\
```

I am new to this and I do not fully understand how things work. Can someone please help me out in understanding what must be going wrong? Is this a version upgrade issue or something?

Thanks!

Contributor guide

Open the contributing guide

Research direction

Start with the Fluentd and CloudWatch agent pod logs shown in the report, then investigate access to the ECS task metadata endpoint for the affected jobs. Compare the current logging setup with the previously working configuration; done means identifying the cause and restoring the jobs' logs in CloudWatch.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, kubernetes
Domain
cloud, observability-sre
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.