aws / aws/containers-roadmap

[ECS/FireLens] [request]: Use Dots in Fluent log tags generated by FireLens

Open
#994 0 comments 4 reactions 0 assignees View on GitHub
ECS Proposed
Dominant language
Shell
Stars
5.4k
Forks
334
PR merge metrics
No merged PRs in 30d

Description

### Community Note

* Please vote on this issue by adding a 👍 [reaction](https://blog.github.com/2016-03-10-add-reactions-to-pull-requests-issues-and-comments/) to the original issue to help the community and maintainers prioritize this request
* Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
* If you are interested in working on this issue or have submitted a pull request, please leave a comment

**Tell us about your request**

For background, how FireLens works is explained here: https://aws.amazon.com/blogs/containers/under-the-hood-firelens-for-amazon-ecs-tasks/

In the Fluentd and Fluent Bit community, it is standard practice to separate the "parts" of a tag with the dot (".") character.

For example, the Fluent Bit rewrite tag character supports a syntax to select a part of a tag delimited with dots: https://docs.fluentbit.io/manual/v/1.4/pipeline/filters/rewrite-tag

If the tag is `aa.bb.cc`, then `$TAG[1]` would be `bb`.

I knew that dot delimiting was a standard practice before I designed FireLens. Yet, I made that tag for container logs delimited with dashes instead `{container name}-firelens-{task ID}`. Several people have pointed out on twitter that this is weird. I am sorry. I do not know why I did that.

In the future, I hope to add a feature in the CloudWatch Logs output so that the log stream or group can be named based on parts of the tag. Imagine you could put in your config:

```
log_stream_format $TAG[0]-logs-$TAG[2]
log_group_format myservice-$TAG[2]
```

Basically, customize the name of the stream/group based on parts of the tag.

To enable that, we need a new field in FireLens to use dots in the tag. This unfortunately probably needs to be an opt-in feature, just because we'd be worried about breaking people who have set up match patterns on the existing tags with dashes.

This is very low priority feature request. There are other things which are very urgently needed for logging in ECS/EKS/Fluent Bit, this is more of a nice-to-have.

**Which service(s) is this request for?**
ECS

Contributor guide

Open the contributing guide

Research direction

Start with the linked FireLens background article and Fluent Bit rewrite-tag documentation to understand the current tag format and its consumers. The issue describes an opt-in dot-delimited tag format for ECS FireLens while preserving existing dash-delimited matching; implementation files and tests are not identified in the payload.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws
Domain
cloud, observability-sre
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.