elastic / elastic/integrations
[kubernetes/container_logs]: Allow creating single input to monitor all log files
- Dominant language
- Handlebars
- Stars
- 333
- Forks
- 647
- Avg merge
- 3d 4h
- Merged PRs (30d)
- 209
Description
### Integration Name
Kubernetes [kubernetes]
### Dataset Name
kubernetes.container_logs
### Integration Version
1.84.1
### Agent Version
9.2.1
### OS Version and Architecture
Darwin 25.2.0
### User Goal
Add an opt-in config option to create one filestream input to monitor the entire cluster. The `paths` would be set to `/var/log/pods/*/*/*.log` by default.
This feature will be opt-in and will NOT be the default.
### Existing Features
The kubernetes.container_logs data stream creates one filestream input per container. For 100s of pod and potentially 1000s of containers, this becomes an issue because each input comes with a baseline memory usage.
This can lead to OOMs and restarts in production environment.
### What did you see?
I manually installed a policy with one filestream input monitoring `/var/log/containers/*.log` as a whole and tested it against the latest version kubernetes integration. That showed remarkable reduction in overall memory usage.
Attaching images:
1. Kubernetes integration 1.84.1.
2. Custom policy with one input and rest of the config same as previous one.
### Anything else?
If any customer wants to switch to this feature, they'd face data duplication as the input id will change. We'll be documenting this caveat.
Contributor guide
Assessment
This issue has not been assessed yet.