aws / aws/containers-roadmap

[ECS] [request]: AWS firelens config missing options for S3 and path

Open
#946 3 comments 25 reactions 0 assignees View on GitHub
ECS Proposed
Dominant language
Shell
Stars
5.4k
Forks
334
PR merge metrics
No merged PRs in 30d

Description

### Community Note

* Please vote on this issue by adding a 👍 [reaction](https://blog.github.com/2016-03-10-add-reactions-to-pull-requests-issues-and-comments/) to the original issue to help the community and maintainers prioritize this request
* Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
* If you are interested in working on this issue or have submitted a pull request, please leave a comment

**Tell us about your request**
On the task-definition page, for firelens configuration for fluentbit, we need additional fields for advanced routing options - currently, the option to change the config to file or S3 isn't available from the AWS console. Some of the fluentbit routing magic lies in the ability to use custom config files with different outputs, but in order to use those features, we need to change the options to file. I've attached two screenshots below to provide some guidance where a good place would be for those new fields and which config values I am referring to

Below is a suggestion for where we can place those extra required fields
![aws_firelens_options](https://user-images.githubusercontent.com/6543872/84688668-01baab80-af40-11ea-8bea-0bce7344a72e.png)

Where in the task-definition file are those fields then? - see screenshot below
![aws_firelens_options2](https://user-images.githubusercontent.com/6543872/84688739-23b42e00-af40-11ea-984d-6b073beed509.png)

**Which service(s) is this request for?**
This could be Fargate, ECS

**Tell us about the problem you're trying to solve. What are you trying to do, and why is it hard?**
Trying to setup fargate containers with firelens using fluentbit as log-router, shipping logs to multi-outputs. In our case, cloudwatch and datadog. In order to route logs to multiple outputs, require a custom config file for fluentbit, and at this stage, the AWS console for creating the task-definition doesn't provide those features to specify S3 or File as config.

**Are you currently working around this issue?**
No solution at this stage - one option we persuing is to create a custom config file, upload that as part of a custom fluentbit image and use the aws cli to create a task-definition with the options set to file

**Additional context**
Also struggled with using the aws cli task-definition setting the fluentbit content to S3 - meaning pull the custom config file from S3 ... the fargate container crash all the time when using a config file hosted on S3 - even after adding the ECS role full access to S3, still failed. That same config file work perfectly fine in a normal fluentbit startup, but placed on S3, and using the task-definition with firelens options set to use S3 never worked. - our last resort will be changing this to file

First created a request on this GitHub page but closed it - makes more sense for the container roadmap - https://github.com/aws/amazon-cloudwatch-logs-for-fluent-bit/issues/52

Contributor guide

Open the contributing guide

Research direction

Start with the ECS task-definition page and the FireLens configuration fields described in the issue, comparing the requested file and S3 options with the AWS CLI task-definition settings. Verify the desired result by confirming a Fargate task can use a custom Fluent Bit config for multiple outputs, including CloudWatch and Datadog; the issue also reports an unresolved S3 failure.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws
Domain
cloud
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.