aws / aws/containers-roadmap

[EKS] [request]: Dynamic creation of Elasticsearch indexes

Open
#1,055 5 comments 13 reactions 0 assignees View on GitHub
EKS Proposed
Dominant language
Shell
Stars
5.4k
Forks
334
PR merge metrics
No merged PRs in 30d

Description

### Community Note

* Please vote on this issue by adding a 👍 [reaction](https://blog.github.com/2016-03-10-add-reactions-to-pull-requests-issues-and-comments/) to the original issue to help the community and maintainers prioritize this request
* Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
* If you are interested in working on this issue or have submitted a pull request, please leave a comment

**Tell us about your request**
To be able to have fluent-bit use the Kubernetes namespaces as an Elasticsearch index name dynamically.

**Which service(s) is this request for?**
EKS

**Tell us about the problem you're trying to solve. What are you trying to do, and why is it hard?**
I am not able to dynamically create Elasticsearch indexes based on the Kubernetes namespace name. Because in our setup the namespace name is customer separated and from a platform perspective we are not aware of the customers that get created, we cannot pre-provision the indexes.

**Are you currently working around this issue?**
We currently have to run multiple fluentd pods just to do the dynamic index creation for us. Because all logs have to go through the fluentd pods they are becoming a bottleneck.

**Additional context**
Ideally it would be as simple as using a variable for the Elasticsearch index name, that gets substituted with the kubernetes namespace name. Perhaps via Tags.

Linked issue: [https://github.com/fluent/fluent-bit/issues/2514](https://github.com/fluent/fluent-bit/issues/2514)

Contributor guide

Open the contributing guide

Research direction

Start with the linked Fluent Bit issue #2514 and review the Elasticsearch output configuration, focusing on the requested variable or tag-based index naming. No repository files or tests are named in this issue. Done means Fluent Bit can create Elasticsearch indexes dynamically from Kubernetes namespace names without requiring multiple Fluentd pods.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, elasticsearch, kubernetes
Domain
cloud, observability-sre
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
28/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.