fluent / fluent/fluent-bit

[filter_kubernetes] enhancement: provide mechanism to exclude containers from fluent bit via annotations

Open
#737 22 comments 29 reactions 1 assignee Claimed by @edsiper View on GitHub
enhancement
Dominant language
C
Stars
8.1k
Forks
2k
Avg merge
4d 20h
Merged PRs (30d)
71

Description

**Problem Statement**
In the current implementation the most common way to get container logs parsed by fluentbit in a kubernetes cluster is to have a filter applied to containers using a log message like the one detailed here:

```
input-kubernetes.conf: |
[INPUT]
Name tail
Tag kube.*
Path /var/log/containers/*.log
Parser docker
DB /var/log/flb_kube.db
```

This represents a bit of a problem with dev containers sometimes polluting our log platform with unwanted logs. We have a select few services we want to include, but the default rule is to exclude.

**Describe the solution you'd like**

A simple way to include logs in the opposite of how #555 was implemented where you can annotate containers with a `fluentbit.io/include: "true"` and the fluentbit daemonset will only pick up these logs.

**Describe alternatives you've considered**

I've updated my Path in the above config to `/var/log/containers/*deployment*.log` and ensured all my deployments I want to aggregate logs for have `deployment` in the name.

**Additional context**

I have a Kubernetes cluster set up with kops 1.10 and used https://github.com/fluent/fluent-bit-kubernetes-logging to set up fluentbit which then forwards to a fluentd service running the logz.io plugin.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.