Docker Log Recursion - Unable to Exclude Fluent Bit's Own Logs Without Container ID
- Dominant language
- C
- Stars
- 8.1k
- Forks
- 2k
- Avg merge
- 4d 16h
- Merged PRs (30d)
- 58
Description
## Problem Description
When using Fluent Bit to collect Docker container logs via the `tail` input plugin with `Docker_Mode On`, Fluent Bit collects its own logs, creating an infinite recursion loop. The only workaround is to exclude logs by container ID, which is not generic and breaks when containers are recreated.
## Expected Behavior
Fluent Bit should be able to exclude its own logs using container metadata (labels, container name, etc.) rather than relying on container-specific IDs.
## Current Limitations
1. **No container metadata fields**: The `tail` input plugin with `Docker_Mode On` only provides `time`, `log`, and `stream` fields. No container metadata (labels, container_name, etc.) is exposed.
2. **No generic exclusion**: The `grep` filter cannot access fields like `$labels['app']`, `$container_name`, or `$docker['container_labels']` because they don't exist in the parsed records.
3. **Forced to use container ID**: The only working exclusion is via `$file ^.*.*$`, which is not portable across deployments.
## Reproduction Steps
1. Run Fluent Bit with Docker log collection:
```yaml
# docker-compose.yaml
services:
fluent-bit:
image: fluent/fluent-bit:latest
volumes:
- /var/lib/docker/containers:/var/lib/docker/containers:ro
- /var/run/docker.sock:/var/run/docker.sock:ro
command: /fluent-bit/bin/fluent-bit -c /fluent-bit/etc/fluent-bit.conf
```
2. Configuration that fails to exclude Fluent Bit's own logs:
```conf
[INPUT]
Name tail
Path /var/lib/docker/containers/*/*.log
Parser docker
Docker_Mode On
[FILTER]
Name grep
Match *
# These don't work:
Exclude $labels['app'] ^fluent-bit$
Exclude $container_name ^fluent-bit$
```
3. Observe recursive logging and infinite loop.
## Current Workaround (Not Acceptable)
```conf
[FILTER]
Name grep
Match *
# Container ID changes on recreation
Exclude $file ^.*abcd1234.*$
```
## Error/Warning Messages
```
[ warn] [env] variable ${labels} is used but not set
[ warn] [env] variable ${container_name} is used but not set
[ warn] [env] variable ${docker['container_labels']} is used but not set
```
## Environment
- **Fluent Bit version**: 4.2.0 (also observed in earlier versions)
- **Docker version**: Any
- **OS**: Linux
## Suggested Solutions
### Option 1: Enhance Docker Mode to Include Metadata
Extend the `tail` input plugin's `Docker_Mode` to add container metadata fields:
- `container_id`, `container_name`, `image`, `labels`, etc.
### Option 2: Add Built-in Self-Exclusion Filter
Add a filter that automatically excludes Fluent Bit's own logs:
```conf
[FILTER]
Name exclude_self
Match *
# Automatically detects and excludes fluent-bit containers
```
### Option 3: Expose Docker API Fields
Make Docker API metadata available via the `docker` filter (which currently doesn't exist in some distributions):
```conf
[FILTER]
Name docker
Match *
# Adds container metadata for filtering
```
### Option 4: Support Label-Based Filtering in Input
Add label filtering directly to the input plugin:
```conf
[INPUT]
Name tail
Path /var/lib/docker/containers/*/*.log
Parser docker
Docker_Mode On
Exclude_Labels app=fluent-bit
```
## Use Case Importance
This is critical for:
1. **Dynamic environments** where container IDs change frequently
2. **CI/CD pipelines** with disposable containers
## Related Issues
- Lack of `docker` filter plugin in some Fluent Bit distributions
- Inability to access container labels for routing decisions
## Additional Context
Users have tried various workarounds:
1. **Lua scripts** to query Docker API (requires additional tools not in minimal images)
2. **External tools** like logspout (adds complexity)
3. **Custom-built Fluent Bit** with docker filter (not standard)
4. **Container ID exclusion** (breaks on container recreation)
This issue prevents Fluent Bit from being a truly generic Docker logging solution and forces users to adopt fragile, deployment-specific configurations.
---
**Impact**: High - Affects all Docker deployments using Fluent Bit for log collection
**Priority**: Critical for production environments
**Workaround Exists**: Yes, but unacceptable (container ID dependent)
Contributor guide
Assessment
This issue has not been assessed yet.