elastic / elastic/elastic-agent

`version_conflict_engine_exception` for Agent metrics in Logstash 9.4.2

Open
#14,860 9 comments 0 reactions 0 assignees View on GitHub
bug Team:Elastic-Agent-Data-Plane
Dominant language
Go
Stars
276
Forks
266
Avg merge
1d 23h
Merged PRs (30d)
312

Description

Versions:
- Elastic Agent: 9.4.2
- Logstash: 9.4.2
- Elasticsearch: 9.4.2

Operating System(s):
- Linux
- Windows

(All agents are Fleet-managed.)

### Summary

When Elastic Agent's metrics collection ("Collect agent metrics") is enabled (in _Agent Policies->Settings->Agent monitoring_), and data is routed through Logstash, documents in .metrics-* data streams are duplicated, resulting in 409 `version_conflict_engine_exception` errors during indexing (Logstash errors).

The issue appears to be specific to Elastic Agent metrics (not logs). Disabling agent metrics collection immediately stops both the metrics traffic and the version conflict errors.

### Environment
- Fleet-managed Elastic Agents (Deployed on several Windows and Linux Hosts)
- Logstash used as output between Elastic Agent and Elasticsearch
- Metrics and logs use the same Logstash output

### Steps to Reproduce
Deploy a Fleet-managed Elastic Agent (tested on both Linux and Windows).
Configure the agent to send data through Logstash (Logstash Output).
**Enable Collect agent metrics in the Fleet policy.**

### Expected Behavior
Agent metrics documents should be indexed once without conflicts and errors.

### Actual Behavior
Documents in `.metrics-*` data streams seem to get indexed multiple times, resulting in **409** `version_conflict_engine_exception ` errors.

The conflicts occur consistently while agent metrics collection is enabled.

### Additional Findings
Disabling Collect agent metrics immediately stops:
`.metrics-*` traffic
409 `version_conflict_engine_exception` errors

The issue is reproducible with minimal Fleet policies and no additional integrations enabled.
Agent diagnostics confirm that logs and metrics share the same Logstash output configuration.

The behavior occurs with both:
- a single Logstash host
- multiple Logstash hosts with loadbalance: true in Elastic Agent configs

Other logs and integration data do not appear to exhibit the same duplication behavior.
The issue appears isolated to Elastic Agent metrics data streams.

### Questions
Is this a known issue affecting Elastic Agent metrics when routed through Logstash?
Is this expected behaviour for metrics?

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the issue with a Fleet-managed Elastic Agent sending metrics through Logstash, using the minimal policy described in the report. Inspect Agent diagnostics and Logstash indexing errors to trace why .metrics-* documents are duplicated; done means each metrics document is indexed once without 409 version_conflict_engine_exception errors.

Written by the indexing model from the issue text.

Assessment

Tech stack
elasticsearch
Domain
observability-sre
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.