[EKS] [request]: Add EKS Fargate fleuntd output or similiar
- Dominant language
- Shell
- Stars
- 5.4k
- Forks
- 334
- PR merge metrics
- No merged PRs in 30d
Description
### Community Note
* Please vote on this issue by adding a 👍 [reaction](https://blog.github.com/2016-03-10-add-reactions-to-pull-requests-issues-and-comments/) to the original issue to help the community and maintainers prioritize this request
* Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
* If you are interested in working on this issue or have submitted a pull request, please leave a comment
**Tell us about your request**
What do you want us to build?
Fargate eks node doesn't support fluentd output, so we have to build a log system for fargate
https://docs.aws.amazon.com/eks/latest/userguide/fargate-logging.html
**Which service(s) is this request for?**
This could be Fargate, ECS, EKS, ECR
EKS fargate node
**Tell us about the problem you're trying to solve. What are you trying to do, and why is it hard?**
What outcome are you trying to achieve, ultimately, and why is it hard/impossible to do right now? What is the impact of not having this problem solved? The more details you can provide, the better we'll be able to understand and solve the problem.
We have centerlize log aggregator with fluentd but because fargate doesn't have ds (deamonsets) and it is using fluent-bit and we can't shift the log directly using fluentl-bit
**Are you currently working around this issue?**
How are you currently solving this problem?
Ship to another cloudwatch or ES, this cause extra burden to maintain/create extra system because AWS doesn't support this. And the log is not longer centerise and allows all different users to search it.
**Additional context**
Anything else we should know?
We are using normal VM as EKS node and everything is fine with all other system. We learn about fargate and we have particular use case (report generation job) that would best fit with fargate.
We want to
1. match https://github.com/fabric8io/fluent-plugin-kubernetes_metadata_filter with fargate fluentd
2. plus able to add some of our field (cluster)
3. and also tag it as we wanted "eks.[env].[namespace].[pod_name]"
Provide full example how we can use it, particularly we would need to translate everything from fluent to fluent-bit.
* For instance, we want to add "cluster" "fleuntd field" in all the logs, so we know where it is from in multi cluster setup.
* Also we would like to learn how to easily parse the kubernetes meta-data like pod_name and namespace so that it is matching our other system tagging standard. For example we want the tag to be "eks.[env].[namespace].[pod_name]"
Most example are lagging in case of tag parsing
https://github.com/awsdocs/amazon-eks-user-guide/blob/master/doc_source/fargate-logging.md
https://github.com/aws-samples/amazon-eks-fluent-logging-examples/blob/mainline/examples/fluent-bit-appmesh/fluent-bit/fluent-bit-configmap.yaml
https://medium.com/@elfakharany/how-to-use-the-fluent-bit-rewrite-tag-plugin-with-the-kubernetes-plugin-for-log-parsing-5934a85ce9e
**Attachments**
If you think you might have additional information that you'd like to include via an attachment, please do - we'll take a look. (Remember to remove any personally-identifiable information.)
Contributor guide
Research direction
Start with the EKS Fargate logging documentation and the referenced fluent-bit-appmesh/fluent-bit/fluent-bit-configmap.yaml example, then compare the requested metadata, cluster field, and tag format with the linked fluentd metadata filter. Done would require an agreed Fluent Bit approach and a complete usage example for centralized Fargate logging; no repository files or tests are identified in the issue.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, kubernetes
- Domain
- cloud, observability
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100