[ECS] [request]: AWS firelens config missing options for S3 and path
- Dominant language
- Shell
- Stars
- 5.4k
- Forks
- 334
- PR merge metrics
- No merged PRs in 30d
Description
### Community Note
* Please vote on this issue by adding a 👍 [reaction](https://blog.github.com/2016-03-10-add-reactions-to-pull-requests-issues-and-comments/) to the original issue to help the community and maintainers prioritize this request
* Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
* If you are interested in working on this issue or have submitted a pull request, please leave a comment
**Tell us about your request**
On the task-definition page, for firelens configuration for fluentbit, we need additional fields for advanced routing options - currently, the option to change the config to file or S3 isn't available from the AWS console. Some of the fluentbit routing magic lies in the ability to use custom config files with different outputs, but in order to use those features, we need to change the options to file. I've attached two screenshots below to provide some guidance where a good place would be for those new fields and which config values I am referring to
Below is a suggestion for where we can place those extra required fields

Where in the task-definition file are those fields then? - see screenshot below

**Which service(s) is this request for?**
This could be Fargate, ECS
**Tell us about the problem you're trying to solve. What are you trying to do, and why is it hard?**
Trying to setup fargate containers with firelens using fluentbit as log-router, shipping logs to multi-outputs. In our case, cloudwatch and datadog. In order to route logs to multiple outputs, require a custom config file for fluentbit, and at this stage, the AWS console for creating the task-definition doesn't provide those features to specify S3 or File as config.
**Are you currently working around this issue?**
No solution at this stage - one option we persuing is to create a custom config file, upload that as part of a custom fluentbit image and use the aws cli to create a task-definition with the options set to file
**Additional context**
Also struggled with using the aws cli task-definition setting the fluentbit content to S3 - meaning pull the custom config file from S3 ... the fargate container crash all the time when using a config file hosted on S3 - even after adding the ECS role full access to S3, still failed. That same config file work perfectly fine in a normal fluentbit startup, but placed on S3, and using the task-definition with firelens options set to use S3 never worked. - our last resort will be changing this to file
First created a request on this GitHub page but closed it - makes more sense for the container roadmap - https://github.com/aws/amazon-cloudwatch-logs-for-fluent-bit/issues/52
Contributor guide
Research direction
Start with the ECS task-definition page and the FireLens configuration fields described in the issue, comparing the requested file and S3 options with the AWS CLI task-definition settings. Verify the desired result by confirming a Fargate task can use a custom Fluent Bit config for multiple outputs, including CloudWatch and Datadog; the issue also reports an unresolved S3 failure.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws
- Domain
- cloud
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100