aws / aws/containers-roadmap

[ECS] [Fargate]: Partial json logs concatenation with Fluentbit

Open
#2,110 0 comments 39 reactions 0 assignees View on GitHub
ECS Fargate Proposed
Dominant language
Shell
Stars
5.4k
Forks
334
PR merge metrics
No merged PRs in 30d

Description

### Community Note

* Please vote on this issue by adding a 👍 [reaction](https://blog.github.com/2016-03-10-add-reactions-to-pull-requests-issues-and-comments/) to the original issue to help the community and maintainers prioritize this request
* Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
* If you are interested in working on this issue or have submitted a pull request, please leave a comment

**Tell us about your request**
I want to have multi-line json logging supported with the built-in Fluent Bit logging on ECS/Fargate.
One option would be to make proposed config available by simply referencing the path like
```
"config-file-value": "/container-json.conf"
```
instead of building custom image for that. It's not ideal we need to maintain additional image to make it work.

**Which service(s) is this request for?**
ECS Fargate

**Tell us about the problem you're trying to solve. What are you trying to do, and why is it hard?**
Container logs are being splitted into 16kb chunks with addition of `partial` attributes. This prevents centralized logging system to display the logs properly.

**Are you currently working around this issue?**
I have built a custom image based on official AWS fluentbit image and added custom config for parsing.

`Dockerfile`
```
FROM public.ecr.aws/aws-observability/aws-for-fluent-bit:stable

ADD extra.conf /extra.conf
```

`extra.conf` based on https://docs.fluentbit.io/manual/pipeline/filters/multiline-stacktrace#docker-partial-message-use-case
```
[SERVICE]
Parsers_File /fluent-bit/parsers/parsers.conf
Flush 1
Grace 30

[FILTER]
name multiline
match *
multiline.key_content log
mode partial_message

[FILTER]
Name parser
Match *
Key_Name log
Parser json
Reserve_Data True
```

Then in task definiton we have container defined:
```
{
"essential": true,
"image": "***.dkr.ecr.us-east-1.amazonaws.com/***/fluentbit:stable",
"name": "log-router",
"firelensConfiguration": {
"type": "fluentbit",
"options": {
"enable-ecs-log-metadata": "true",
"config-file-type": "file",
"config-file-value": "/extra.conf"
}
}
},
```

**Additional context**
https://github.com/aws/containers-roadmap/issues/1326
https://github.com/aws/aws-for-fluent-bit/issues/100
https://github.com/fluent/fluent-bit/issues/821

**Attachments**
Splitted logs
![Screenshot 2023-08-10 at 11 03 33](https://github.com/aws/containers-roadmap/assets/8074860/a1c18cc1-9bb6-456f-8588-a03a3db17fb9)
![Screenshot 2023-08-10 at 11 04 19](https://github.com/aws/containers-roadmap/assets/8074860/9f48b6c2-a444-44fb-bb50-c6a8b77add60)

Merged logs
![Screenshot 2023-08-10 at 11 03 53](https://github.com/aws/containers-roadmap/assets/8074860/001af819-0fff-4a9e-b7d1-6960839025cc)
![Screenshot 2023-08-10 at 11 04 06](https://github.com/aws/containers-roadmap/assets/8074860/fa7fb8f8-6cce-4557-a592-3170ddf1d5f5)

Contributor guide

Open the contributing guide

Research direction

Start by reading the linked ECS/Fargate and Fluent Bit issues, then review the proposed Dockerfile, extra.conf, and task-definition firelensConfiguration in this report. Confirm how built-in Fluent Bit configuration files are supplied and define done as supporting multiline JSON log merging without maintaining a custom image.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, docker
Domain
cloud, observability
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.