elastic / elastic/integrations

[kubernetes/container_logs]: Allow creating single input to monitor all log files

Open
#17,250 12 comments 1 reaction 0 assignees View on GitHub
Integration:kubernetes Team:Elastic-Agent-Data-Plane Team:obs-ds-hosted-services
Dominant language
Handlebars
Stars
333
Forks
647
Avg merge
3d 4h
Merged PRs (30d)
209

Description

### Integration Name

Kubernetes [kubernetes]

### Dataset Name

kubernetes.container_logs

### Integration Version

1.84.1

### Agent Version

9.2.1

### OS Version and Architecture

Darwin 25.2.0

### User Goal

Add an opt-in config option to create one filestream input to monitor the entire cluster. The `paths` would be set to `/var/log/pods/*/*/*.log` by default.

This feature will be opt-in and will NOT be the default.

### Existing Features

The kubernetes.container_logs data stream creates one filestream input per container. For 100s of pod and potentially 1000s of containers, this becomes an issue because each input comes with a baseline memory usage.
This can lead to OOMs and restarts in production environment.

### What did you see?

I manually installed a policy with one filestream input monitoring `/var/log/containers/*.log` as a whole and tested it against the latest version kubernetes integration. That showed remarkable reduction in overall memory usage.

Attaching images:

1. Kubernetes integration 1.84.1.
Image

2. Custom policy with one input and rest of the config same as previous one.
Image

### Anything else?

If any customer wants to switch to this feature, they'd face data duplication as the input id will change. We'll be documenting this caveat.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.