aws / aws/containers-roadmap

[ECS] [request]: Set Grace to 30 for FireLens Fluent Bit Users

Open
#579 2 comments 37 reactions 0 assignees View on GitHub
ECS Proposed
Dominant language
Shell
Stars
5.4k
Forks
334
PR merge metrics
No merged PRs in 30d

Description

**Tell us about your request**
By default, Fluent Bit only waits 5 seconds after receiving a SIGTERM before it shuts down. This is in contrast to the 30 second grace period that ECS and Fargate provide by default.

**Which service(s) is this request for?**
ECS, Fargate

**Tell us about the problem you're trying to solve. What are you trying to do, and why is it hard?**
Trying to optimize against log loss. By using the full grace period, logs that are in the memory or file buffer have a greater chance of reaching their destination.

**Are you currently working around this issue?**
How are you currently solving this problem?

As of version 1.3.2 of the AWS for Fluent Bit image, we have baked an “optimized” configuration file in the image at the path `/fluent-bit/configs/minimize-log-loss.conf`. You can use it with FireLens by adding the following section to the container definition for your Fluent Bit container:

```
"firelensConfiguration": {
"type": "fluentbit",
"options": {
"config-file-type": "file",
"config-file-value": "/fluent-bit/configs/minimize-log-loss.conf"
}
},
```

This "optimized" config file consists of the following:

```
[SERVICE]
Flush 1
Grace 30
```

I will soon be publishing the results of tests on FireLens which show that at high rates of log throughput using this config improves the chances that all of your logs reach their destination when you task dies.

Data is in this blog: https://aws.amazon.com/blogs/containers/under-the-hood-firelens-for-amazon-ecs-tasks/

Setting Flush to 1 is probably not something ECS needs to do. Users can choose if they want that or not. This feature request is just to set Grace to 30.

Contributor guide

Open the contributing guide

Research direction

Start with the FireLens configuration example and /fluent-bit/configs/minimize-log-loss.conf described in the issue, then read the linked FireLens blog for the shutdown behavior and test evidence. Done means the ECS/Fargate FireLens Fluent Bit default uses Grace 30, without changing the requested Flush behavior, and the effect on buffered log delivery is validated.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws
Domain
cloud
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.