aws / aws/containers-roadmap

[EKS][Fargate] [request]: visibility of errors when enabling logging using EKS Fargate

Open
#2,293 0 comments 1 reaction 0 assignees View on GitHub
EKS Fargate Proposed
Dominant language
Shell
Stars
5.4k
Forks
334
PR merge metrics
No merged PRs in 30d

Description

### Community Note

* Please vote on this issue by adding a 👍 [reaction](https://blog.github.com/2016-03-10-add-reactions-to-pull-requests-issues-and-comments/) to the original issue to help the community and maintainers prioritize this request
* Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
* If you are interested in working on this issue or have submitted a pull request, please leave a comment

**Tell us about your request**
Surfacing of error messages raised by the Fargate-managed Fluent Bit, specifically IAM permissions issues. This could be extended to Fluent Bit configuration, but there are ways to validate this as described in the docs ([ref](https://docs.aws.amazon.com/eks/latest/userguide/fargate-logging.html#:~:text=You%20can%20also%20run%20Fluent%20Bit%20on%20Amazon%20EC2%20using%20the%20desired%20configuration%20to%20troubleshoot%20any%20issues%20that%20arise%20from%20validation.%20Create%20your%20ConfigMap%20using%20one%20of%20the%20following%20examples.)).

**Which service(s) is this request for?**
EKS Fargate

**Tell us about the problem you're trying to solve. What are you trying to do, and why is it hard?**
The only recommendation from AWS documentation for troubleshooting is to [describe the Pod](https://docs.aws.amazon.com/eks/latest/userguide/fargate-logging.html#fargate-logging-troubleshooting). This does not provide any information related to IAM permissions errors.

**Are you currently working around this issue?**
Firstly, set up a dummy Kinesis stream in the same account as the Fargate Pod execution role to ensure it can push logs to the stream.

Secondly, replicate our cross-account logging set-up to verify log messages are reaching each point in the process, which is:
1. Fargate Pod Execution role assuming IAM role in central account used to store logs
2. Central IAM role can push logs to Kinesis stream
3. Lambda gets records from Kinesis for processing
4. Lambda pushes records to OpenSearch

Validate each stage and slowly work through IAM permissions to verify.

This was time consuming, and would be made much quicker if IAM permissions errors were surfaced somewhere. This could be similar to the Pod events described in the troubleshooting section of the docs.

An added complexity was making any changes to our Fluent Bit configuration. If the configuration was incorrect, we'd be in the dark as to what the issue is.

**Additional context**
My team builds and manages EKS clusters in multiple AWS accounts across our organisation for different development teams. We deploy Fluent Bit as a DaemonSet to all clusters, and forward logs to OpenSearch via Kinesis in a central AWS account (cross-account access required).

We're currently migrating to Karpenter, which we've deployed to EKS Fargate support scaling to 0. Naturally, we'd like to forward Karpenter logs to the central log sink too.

We've now got a working solution, but it took a lot of trial and error due to the lack of troubleshooting possible with Fargate-managed logging.

If this is a duplicate, please let me know and I would be happy to close.

**Attachments**
N/A

Contributor guide

Open the contributing guide

Research direction

Start with the EKS Fargate logging documentation, especially the troubleshooting section that directs users to describe the Pod, and review the reported logging path through the Fargate-managed Fluent Bit, IAM roles, Kinesis, Lambda, and OpenSearch. Done means users can find actionable IAM-permission or Fluent Bit configuration errors rather than having to validate each cross-account pipeline stage manually.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, kubernetes
Domain
cloud, observability
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.