aws / aws/containers-roadmap

[Fargate] [request]: Enhance the reliability of FireLens on Fargate

Open
#700 1 comment 80 reactions 0 assignees View on GitHub
ECS Fargate FluentBit Proposed Under consideration
Dominant language
Shell
Stars
5.4k
Forks
334
PR merge metrics
No merged PRs in 30d

Description

### Community Note

* Please vote on this issue by adding a 👍 [reaction](https://blog.github.com/2016-03-10-add-reactions-to-pull-requests-issues-and-comments/) to the original issue to help the community and maintainers prioritize this request
* Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
* If you are interested in working on this issue or have submitted a pull request, please leave a comment

**Tell us about your request**
FireLens has been [demonstrated to be a fairly reliable log solution](https://aws.amazon.com/blogs/containers/under-the-hood-firelens-for-amazon-ecs-tasks/).

That being said, FireLens could go farther on Fargate, become more managed and provide greater assurances of reliability. The Fargate platform has unique challenges because containers are ephemeral. Currently, the FireLens container is just another container in the Task, and when the Task stops is gets a standard 30 second Sigterm to SigKill timeout. Furthermore, in the unlikely case that Fluentd/Bit goes down, all logs would be lost because Fargate containers are ephemeral.

Ideally, AWS could provide two features for FireLens on Fargate to improve reliability:

1. Enable a file buffer for the Fluentd/Bit FireLens container, and restart the container if it goes down. Failures in the FireLens container would not stop a task, and logs would be preserved between stops and re-starts.
2. Build a more robust mechanism than the built in Sigterm-SigKill timeout for the FireLens container. Ideally, after a task stops the FireLens container would be given sufficient time to send all logs/data (up to a reasonable timeout measured in minutes). This might require changes to Fluent Bit. That way, when your task stops, all logs/data would be retrieved. (Note that providing a hard guarantee/promise around reliability is almost certainly impossible).

Contributor guide

Open the contributing guide

Research direction

The issue names no repository files, tests, or implementation entry points. Start by reviewing the linked FireLens overview and the described Fluentd/Fluent Bit buffering and shutdown behavior on Fargate. Done would require AWS-managed reliability improvements for container restarts, file buffering, and graceful log delivery, so this needs platform-level design rather than a newcomer-sized repository change.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws
Domain
cloud
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.