[EKS] [request]: Dynamic creation of Elasticsearch indexes
- Dominant language
- Shell
- Stars
- 5.4k
- Forks
- 334
- PR merge metrics
- No merged PRs in 30d
Description
### Community Note
* Please vote on this issue by adding a 👍 [reaction](https://blog.github.com/2016-03-10-add-reactions-to-pull-requests-issues-and-comments/) to the original issue to help the community and maintainers prioritize this request
* Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
* If you are interested in working on this issue or have submitted a pull request, please leave a comment
**Tell us about your request**
To be able to have fluent-bit use the Kubernetes namespaces as an Elasticsearch index name dynamically.
**Which service(s) is this request for?**
EKS
**Tell us about the problem you're trying to solve. What are you trying to do, and why is it hard?**
I am not able to dynamically create Elasticsearch indexes based on the Kubernetes namespace name. Because in our setup the namespace name is customer separated and from a platform perspective we are not aware of the customers that get created, we cannot pre-provision the indexes.
**Are you currently working around this issue?**
We currently have to run multiple fluentd pods just to do the dynamic index creation for us. Because all logs have to go through the fluentd pods they are becoming a bottleneck.
**Additional context**
Ideally it would be as simple as using a variable for the Elasticsearch index name, that gets substituted with the kubernetes namespace name. Perhaps via Tags.
Linked issue: [https://github.com/fluent/fluent-bit/issues/2514](https://github.com/fluent/fluent-bit/issues/2514)
Contributor guide
Research direction
Start with the linked Fluent Bit issue #2514 and review the Elasticsearch output configuration, focusing on the requested variable or tag-based index naming. No repository files or tests are named in this issue. Done means Fluent Bit can create Elasticsearch indexes dynamically from Kubernetes namespace names without requiring multiple Fluentd pods.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, elasticsearch, kubernetes
- Domain
- cloud, observability-sre
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 28/100