kubernetes: remove need for kube_tag_prefix configuration
- Dominant language
- C
- Stars
- 8.1k
- Forks
- 2k
- Avg merge
- 4d 16h
- Merged PRs (30d)
- 58
Description
**Is your feature request related to a problem? Please describe.**
The Kubernetes filter requires `kube_tag_prefix` to be correctly configured to match the `tail` input plugin configuration. This is so that it then strips the path from the tag to leave just the filename of the log file so we can extract the relevant information to query the K8S API server for the pod metadata.
There are lots of problems occurring with this simple misconfiguration as it is not obvious it is required or when it goes wrong - separately see https://github.com/fluent/fluent-bit/issues/6551 for exposing the error better.
The default assumes `kube.*` is expanded in the `tail` plugin to `kube.var.log.containers.` so any change in the tail plugin breaks this or if the logs are on a different path.
**Describe the solution you'd like**
Keep track of the actual filename directly in the metadata for the record and use that, remove the need for any coordination of configuration between plugins.
Potentially we could also then support other inputs too, e.g. they just need to set this metadata to support the query.
**Describe alternatives you've considered**
**Additional context**
This problem comes up often enough in various contexts (e.g. https://github.com/fluent/fluent-bit/issues/8760) that I wrote a post on it: https://calyptia.com/blog/kubernetes-metadata-enrichment-with-fluent-bit-with-troubleshooting-tips
Contributor guide
Assessment
This issue has not been assessed yet.