elastic / elastic/elastic-agent
Duplicate data collection between generic inputs and specific integrations
- Dominant language
- Go
- Stars
- 275
- Forks
- 264
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 298
Description
Today, we have generic inputs like `Container Logs` or `Custom Logs`. Based on dynamic variables and various providers we can also set this generic inputs to collect logs from all of the log files in a specific path.
For example in Kubernetes (or similarly in Docker) we can set the path like `/var/log/containers/*${kubernetes.container.id}.log` and we will have the input to be set for all the containers based on their `container.id` value.
In addition we can specify
```
- name: nginx
type: nginx/logs
use_output: default
data_stream:
namespace: default
streams:
- data_stream:
dataset: nginx.access
type: logs
paths:
- '/var/log/containers/*${kubernetes.container.id}.log'
condition: ${kubernetes.labels.app} == 'nginx'
```
which is an input based on a condition for `nginx`.
In the end we will have this Pod's logs to be collected twice, from the generic `Container Logs` input and one for the specific one for `Nginx`.
cc: @mukeshelastic
Contributor guide
Assessment
This issue has not been assessed yet.