elastic / elastic/beats

Autodiscover causes `filestream input with ID 'xxx' already exists` when used with Nignx or other modules that enables multiple filesets

Open
#44,443 5 comments 2 reactions 0 assignees View on GitHub
bug Team:Elastic-Agent-Data-Plane
Dominant language
Go
Stars
12.7k
Forks
5k
Avg merge
2d 15m
Merged PRs (30d)
385

Description

When using Filebeat's Nginx module with autodiscover, all modes (hints or template) and providers (Kubernetes, Docker etc), the following error appear in the logs:

```
filestream input with ID 'xxx' already exists, this will lead to data duplication, please use a different ID
```

- For Filebeat >= 9.0, the Filestream input will fail to start
- For Filebeat < 9.0, the Filestream input starts, but it does not register its metrics and there can be data duplication

## Cause
### TL;DR
The Nginx module starts two inputs per container: one for access logs and another for error logs, however both inputs use the same ID provided by the autodiscover.

### The long version
Both forms of autodiscover (hints or template) provide a "default pod template", for hints there is a hardcoded template that can be overriten by `hints.default_config` and for template, it is always provided in the configuration. Those templates only allow for one form for the input ID and there is no way to use variables from the module to add the fileset name, which always leads to two inputs being rendered with the same ID if both filesets are enabled (which is very common).

Replacing the configuration template (an YAML file located at `module/nginx//config/.yml`) for the filesets does not work. First the autodiscover renders its template, adds the module information, then passes this to Filebeat's input loader. The input loader reads the module/fileset configuration template, merges it with the configuration it received, then starts the input. However the input loader does not have access to the autodiscover variables, which prevents us from adding the container ID, always unique, to the input ID.

When hints based autodiscover is enable, the default configuration template is:
https://github.com/elastic/beats/blob/9382cc20546bc60fe703323ff83e258020b1d2ff/filebeat/autodiscover/builder/hints/config.go#L29-L57

# Workaround
Currently, there are some ways to work around the issue:
1. Use the autodiscover with template + modules (e.g: Nginx)
2. Standalone Elastic-Agent hints autodiscover.

## 1. Autodiscover with template + Ngixn module
For that the following configuration can be used
```yaml
filebeat.autodiscover:
providers:
- type: kubernetes
node: ${NODE_NAME}
templates:
- condition:
equals:
kubernetes.annotations.co.elastic.nginx.fileset: "access/error"
config:
- module: nginx
access:
input:
type: filestream
id: nginx-access-${data.kubernetes.container.id}
paths:
- "/var/log/containers/*-${data.kubernetes.container.id}.log"
parsers:
- container:
format: "auto"
stream: "stdout"
prospector.scanner:
symlinks: true
fingerprint.enabled: true
file_identity.fingerprint: ~
error:
input:
type: filestream
id: nginx-error-${data.kubernetes.container.id}
paths:
- "/var/log/containers/*-${data.kubernetes.container.id}.log"
parsers:
- container:
format: "auto"
stream: "stderr"
- multiline:
type: pattern
pattern: '^\d{4}\/\d{2}\/\d{2} '
negate: true
match: after
prospector.scanner:
symlinks: true
fingerprint.enabled: true
file_identity.fingerprint: ~
- condition:
equals:
kubernetes.annotations.co.elastic.nginx.fileset: "ingress_controller"
config:
- module: nginx
ingress_controller:
input:
type: filestream
id: nginx-ingress_controller-${data.kubernetes.container.id}
paths:
- "/var/log/containers/*-${data.kubernetes.container.id}.log"
parsers:
- container:
format: "auto"
stream: "all"
prospector.scanner:
symlinks: true
fingerprint.enabled: true
file_identity.fingerprint: ~
```

Once Filebeat is configured, the pods need to be annotated with a **new annotation**:
- `co.elastic.nginx.fileset: "access/error"` For the `access` and `error` filesets
- `co.elastic.nginx.fileset: "ingress_controller"` for Nginx ingress controller

Here is an example manifest that deploys an Nginx with the acces/error fileset enabled.
```yaml
apiVersion: v1
kind: Pod
metadata:
name: nginx-pod
labels:
app: nginx-pod
annotations:
# co.elastic.nginx.fileset: "ingress_controller"
co.elastic.nginx.fileset: "access/error"
spec:
containers:
- name: nginx
image: nginx:latest
```

## 2. Standalone Elastic-Agent whith hints based autodiscover
There is some [documentation](https://www.elastic.co/docs/reference/fleet/hints-annotations-autodiscovery#_available_packages_that_support_hints_autodiscovery) with examples for Redis and links for all the [supported templates](https://github.com/elastic/elastic-agent/tree/main/deploy/kubernetes/elastic-agent-standalone/templates.d). Before starting to ingest data, it is necessary to install assets from the [Nginx integration](https://www.elastic.co/docs/reference/integrations/nginx/), no need to add it to a policy.

To deploy the Elastic-Agent, use the [manifest](https://raw.githubusercontent.com/elastic/elastic-agent/v9.0.0/deploy/kubernetes/elastic-agent-standalone-kubernetes.yaml) (link for 9.0.0) from [our docs](https://www.elastic.co/docs/reference/fleet/running-on-kubernetes-standalone), making sure to enable hints in `elastic-agent.yml`:

```yaml
providers.kubernetes:
node: ${NODE_NAME}
scope: node
hints.enabled: true
hints.default_container_logs: false
```

And also uncommenting, from the manifest, the init container responsible for downloading the integration templates. The section to be uncommented looks like this:
```yaml
initContainers:
- name: k8s-templates-downloader
image: docker.elastic.co/elastic-agent/elastic-agent:9.0.0
command: ['bash']
args:
- -c
- >-
mkdir -p /etc/elastic-agent/inputs.d &&
curl -sL https://github.com/elastic/elastic-agent/archive/9.0.tar.gz | tar xz -C /etc/elastic-agent/inputs.d --strip=5 "elastic-agent-9.0/deploy/kubernetes/elastic-agent-standalone/templates.d"
volumeMounts:
- name: external-inputs
mountPath: /etc/elastic-agent/inputs.d
```

Contributor guide

Open the contributing guide

Research direction

Start with the hints default template in filebeat/autodiscover/builder/hints/config.go and the Nginx fileset templates under module/nginx//config/. Trace how autodiscover output reaches the Filebeat input loader, which the issue says cannot access autodiscover variables. Done means access and error inputs can run together without duplicate IDs, input startup failures, missing metrics, or data duplication.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, yaml
Domain
observability-sre
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.