Autodiscover causes `filestream input with ID 'xxx' already exists` when used with Nignx or other modules that enables multiple filesets
- Dominant language
- Go
- Stars
- 12.7k
- Forks
- 5k
- Avg merge
- 2d 15m
- Merged PRs (30d)
- 385
Description
When using Filebeat's Nginx module with autodiscover, all modes (hints or template) and providers (Kubernetes, Docker etc), the following error appear in the logs:
```
filestream input with ID 'xxx' already exists, this will lead to data duplication, please use a different ID
```
- For Filebeat >= 9.0, the Filestream input will fail to start
- For Filebeat < 9.0, the Filestream input starts, but it does not register its metrics and there can be data duplication
## Cause
### TL;DR
The Nginx module starts two inputs per container: one for access logs and another for error logs, however both inputs use the same ID provided by the autodiscover.
### The long version
Both forms of autodiscover (hints or template) provide a "default pod template", for hints there is a hardcoded template that can be overriten by `hints.default_config` and for template, it is always provided in the configuration. Those templates only allow for one form for the input ID and there is no way to use variables from the module to add the fileset name, which always leads to two inputs being rendered with the same ID if both filesets are enabled (which is very common).
Replacing the configuration template (an YAML file located at `module/nginx//config/.yml`) for the filesets does not work. First the autodiscover renders its template, adds the module information, then passes this to Filebeat's input loader. The input loader reads the module/fileset configuration template, merges it with the configuration it received, then starts the input. However the input loader does not have access to the autodiscover variables, which prevents us from adding the container ID, always unique, to the input ID.
When hints based autodiscover is enable, the default configuration template is:
https://github.com/elastic/beats/blob/9382cc20546bc60fe703323ff83e258020b1d2ff/filebeat/autodiscover/builder/hints/config.go#L29-L57
# Workaround
Currently, there are some ways to work around the issue:
1. Use the autodiscover with template + modules (e.g: Nginx)
2. Standalone Elastic-Agent hints autodiscover.
## 1. Autodiscover with template + Ngixn module
For that the following configuration can be used
```yaml
filebeat.autodiscover:
providers:
- type: kubernetes
node: ${NODE_NAME}
templates:
- condition:
equals:
kubernetes.annotations.co.elastic.nginx.fileset: "access/error"
config:
- module: nginx
access:
input:
type: filestream
id: nginx-access-${data.kubernetes.container.id}
paths:
- "/var/log/containers/*-${data.kubernetes.container.id}.log"
parsers:
- container:
format: "auto"
stream: "stdout"
prospector.scanner:
symlinks: true
fingerprint.enabled: true
file_identity.fingerprint: ~
error:
input:
type: filestream
id: nginx-error-${data.kubernetes.container.id}
paths:
- "/var/log/containers/*-${data.kubernetes.container.id}.log"
parsers:
- container:
format: "auto"
stream: "stderr"
- multiline:
type: pattern
pattern: '^\d{4}\/\d{2}\/\d{2} '
negate: true
match: after
prospector.scanner:
symlinks: true
fingerprint.enabled: true
file_identity.fingerprint: ~
- condition:
equals:
kubernetes.annotations.co.elastic.nginx.fileset: "ingress_controller"
config:
- module: nginx
ingress_controller:
input:
type: filestream
id: nginx-ingress_controller-${data.kubernetes.container.id}
paths:
- "/var/log/containers/*-${data.kubernetes.container.id}.log"
parsers:
- container:
format: "auto"
stream: "all"
prospector.scanner:
symlinks: true
fingerprint.enabled: true
file_identity.fingerprint: ~
```
Once Filebeat is configured, the pods need to be annotated with a **new annotation**:
- `co.elastic.nginx.fileset: "access/error"` For the `access` and `error` filesets
- `co.elastic.nginx.fileset: "ingress_controller"` for Nginx ingress controller
Here is an example manifest that deploys an Nginx with the acces/error fileset enabled.
```yaml
apiVersion: v1
kind: Pod
metadata:
name: nginx-pod
labels:
app: nginx-pod
annotations:
# co.elastic.nginx.fileset: "ingress_controller"
co.elastic.nginx.fileset: "access/error"
spec:
containers:
- name: nginx
image: nginx:latest
```
## 2. Standalone Elastic-Agent whith hints based autodiscover
There is some [documentation](https://www.elastic.co/docs/reference/fleet/hints-annotations-autodiscovery#_available_packages_that_support_hints_autodiscovery) with examples for Redis and links for all the [supported templates](https://github.com/elastic/elastic-agent/tree/main/deploy/kubernetes/elastic-agent-standalone/templates.d). Before starting to ingest data, it is necessary to install assets from the [Nginx integration](https://www.elastic.co/docs/reference/integrations/nginx/), no need to add it to a policy.
To deploy the Elastic-Agent, use the [manifest](https://raw.githubusercontent.com/elastic/elastic-agent/v9.0.0/deploy/kubernetes/elastic-agent-standalone-kubernetes.yaml) (link for 9.0.0) from [our docs](https://www.elastic.co/docs/reference/fleet/running-on-kubernetes-standalone), making sure to enable hints in `elastic-agent.yml`:
```yaml
providers.kubernetes:
node: ${NODE_NAME}
scope: node
hints.enabled: true
hints.default_container_logs: false
```
And also uncommenting, from the manifest, the init container responsible for downloading the integration templates. The section to be uncommented looks like this:
```yaml
initContainers:
- name: k8s-templates-downloader
image: docker.elastic.co/elastic-agent/elastic-agent:9.0.0
command: ['bash']
args:
- -c
- >-
mkdir -p /etc/elastic-agent/inputs.d &&
curl -sL https://github.com/elastic/elastic-agent/archive/9.0.tar.gz | tar xz -C /etc/elastic-agent/inputs.d --strip=5 "elastic-agent-9.0/deploy/kubernetes/elastic-agent-standalone/templates.d"
volumeMounts:
- name: external-inputs
mountPath: /etc/elastic-agent/inputs.d
```
Contributor guide
Research direction
Start with the hints default template in filebeat/autodiscover/builder/hints/config.go and the Nginx fileset templates under module/nginx//config/. Trace how autodiscover output reaches the Filebeat input loader, which the issue says cannot access autodiscover variables. Done means access and error inputs can run together without duplicate IDs, input startup failures, missing metrics, or data duplication.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, yaml
- Domain
- observability-sre
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100