[EKS] [Fargate]: have fluentbit kubernetes filter honor requests for namespace metadata
- Dominant language
- Shell
- Stars
- 5.4k
- Forks
- 334
- PR merge metrics
- No merged PRs in 30d
Description
### Community Note
* Please vote on this issue by adding a 👍 [reaction](https://blog.github.com/2016-03-10-add-reactions-to-pull-requests-issues-and-comments/) to the original issue to help the community and maintainers prioritize this request
* Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
* If you are interested in working on this issue or have submitted a pull request, please leave a comment
**Tell us about your request**
What do you want us to build?
> Have aws-for-fluent-bit attach `kubernetes_namespace` object to log records when kubernetes filter contains `Namespace_Labels` or `Namespace_Annotations` options set.
**Which service(s) is this request for?**
> Fargate on EKS
**Tell us about the problem you're trying to solve. What are you trying to do, and why is it hard?**
> Kubernetes pod metadata allows us to build a fluentbit configuration that drops log records for pods with a target annotation or label. I am working on a use case that requires that we toggle logging on and off for all workload resources of a target namespace.
> With access to metadata at the namespace level, we could add filters in the fluentbit configuration to drop records when target label/annotation is found at the namespace level which would greatly simplify the toggle of logging for a namespace (if we rely on pod metadata only, toggle would require editing the drop label/annotation for all pods in the namespace which is expensive and error prone)
**Are you currently working around this issue?**
How are you currently solving this problem?
> I am currently studying alternatives to namespace metadata in order to make this design work well but so far I did not manage to find a work around
**Additional context**
Anything else we should know?
**Attachments**
If you think you might have additional information that you'd like to include via an attachment, please do - we'll take a look. (Remember to remove any personally-identifiable information.)
I have enabled process logs for the fluentbit running on Fargate and I see its fluentbit version is 4.0.1. Namespace metadata is a feature introduced in fluentbit 3.0 so I would expect this to work
```
{ "log": "Fluent Bit v4.0.1" }
--
{ "log": "[2025/10/21 14:38:43] [ info] [fluent bit] version=4.0.1, commit=, pid=" }
```
Here is a minimum example of what a configmap to reproduce this looks like
```
kind: Namespace
apiVersion: v1
metadata:
name: aws-observability
labels:
aws-observability: enabled
---
apiVersion: v1
kind: ConfigMap
metadata:
name: aws-logging
namespace: aws-observability
data:
flb_log_cw: 'true' # <- make sure pod execution role has cloudwatch log permissions
filters.conf: |
[FILTER]
Name kubernetes
Match kube.*
Merge_Log On
Kube_Meta_Cache_TTL 30s
Namespace_Labels On # <- should attach namespace labels map to "kubernetes_namespace" object
Namespace_Annotations On # <- should attach namespace annotations map to "kubernetes_namespace" object
Kube_Meta_Namespace_Cache_TTL 30s
output.conf: |
[OUTPUT]
Name cloudwatch
Match kube.*
region
log_group_name fluent-bit-cloudwatch
log_stream_prefix from-fluent-bit-
auto_create_group true
```
log output in cloudwatch destination has no `kubernetes_namespace` object and looks similar to the following:
```
{
"kubernetes": {
"annotations": {
"CapacityProvisioned": "0.25vCPU 0.5GB",
"Logging": "LoggingEnabled”,
...
},
"container_image": “",
"container_name": “",
"host": "fargate-ip-..compute.internal",
"labels": {
"eks.amazonaws.com/fargate-profile": “",
"pod-template-hash": “",
...
},
"namespace_name": “",
"pod_id": “",
"pod_name": “-668cc5567b-q5w5z"
},
"log": “"
}
```
Especially because we seem to be using the latest fluentbit major as the base for this aws fork, I think it would be great if we supported this namespace feature. I am sure it is useful for other use cases out there.
Contributor guide
Research direction
The issue names no repository files, tests, or implementation entry points. Start by locating the aws-for-fluent-bit Kubernetes filter integration and its handling of Namespace_Labels and Namespace_Annotations, then determine how Fargate supplies namespace metadata. Done means configurations using those options attach a kubernetes_namespace object with the requested metadata and have coverage for the provided reproduction.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- kubernetes
- Domain
- cloud, observability-sre
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100