[ECS] [Fargate]: Partial json logs concatenation with Fluentbit
- Dominant language
- Shell
- Stars
- 5.4k
- Forks
- 334
- PR merge metrics
- No merged PRs in 30d
Description
### Community Note
* Please vote on this issue by adding a 👍 [reaction](https://blog.github.com/2016-03-10-add-reactions-to-pull-requests-issues-and-comments/) to the original issue to help the community and maintainers prioritize this request
* Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
* If you are interested in working on this issue or have submitted a pull request, please leave a comment
**Tell us about your request**
I want to have multi-line json logging supported with the built-in Fluent Bit logging on ECS/Fargate.
One option would be to make proposed config available by simply referencing the path like
```
"config-file-value": "/container-json.conf"
```
instead of building custom image for that. It's not ideal we need to maintain additional image to make it work.
**Which service(s) is this request for?**
ECS Fargate
**Tell us about the problem you're trying to solve. What are you trying to do, and why is it hard?**
Container logs are being splitted into 16kb chunks with addition of `partial` attributes. This prevents centralized logging system to display the logs properly.
**Are you currently working around this issue?**
I have built a custom image based on official AWS fluentbit image and added custom config for parsing.
`Dockerfile`
```
FROM public.ecr.aws/aws-observability/aws-for-fluent-bit:stable
ADD extra.conf /extra.conf
```
`extra.conf` based on https://docs.fluentbit.io/manual/pipeline/filters/multiline-stacktrace#docker-partial-message-use-case
```
[SERVICE]
Parsers_File /fluent-bit/parsers/parsers.conf
Flush 1
Grace 30
[FILTER]
name multiline
match *
multiline.key_content log
mode partial_message
[FILTER]
Name parser
Match *
Key_Name log
Parser json
Reserve_Data True
```
Then in task definiton we have container defined:
```
{
"essential": true,
"image": "***.dkr.ecr.us-east-1.amazonaws.com/***/fluentbit:stable",
"name": "log-router",
"firelensConfiguration": {
"type": "fluentbit",
"options": {
"enable-ecs-log-metadata": "true",
"config-file-type": "file",
"config-file-value": "/extra.conf"
}
}
},
```
**Additional context**
https://github.com/aws/containers-roadmap/issues/1326
https://github.com/aws/aws-for-fluent-bit/issues/100
https://github.com/fluent/fluent-bit/issues/821
**Attachments**
Splitted logs


Merged logs


Contributor guide
Research direction
Start by reading the linked ECS/Fargate and Fluent Bit issues, then review the proposed Dockerfile, extra.conf, and task-definition firelensConfiguration in this report. Confirm how built-in Fluent Bit configuration files are supplied and define done as supporting multiline JSON log merging without maintaining a custom image.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, docker
- Domain
- cloud, observability
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100