elastic / elastic/beats

[Metricbeat][Elasticsearch][ingest_pipeline] In x-pack mode, data is sent to the wrong target

Open
#41,850 7 comments 0 reactions 0 assignees View on GitHub
Stalled Team:Monitoring
Dominant language
Go
Stars
12.7k
Forks
5k
Avg merge
2d 15m
Merged PRs (30d)
385

Description

When running in x-pack mode, it is expected that all data from the Elasticsearch module should land in the `.monitoring-*` data stream, but the `ingest_pipeline` metricset doesn't [override](https://github.com/elastic/beats/blob/main/metricbeat/module/elasticsearch/cluster_stats/data.go#L318-L323) the target data stream before the events are shipped, leading them to land in `metricbeat-*`.

Fixing this issue in the Beats code is trivial, but it will also require us to update the Elasticsearch Ingest Pipeline [dashboard](https://github.com/elastic/integrations/blob/main/packages/elasticsearch/kibana/dashboard/elasticsearch-metrics-ingest-pipelines.json) found in the Elasticsearch integration.

Contributor guide

Open the contributing guide

Research direction

Start at metricbeat/module/elasticsearch/cluster_stats/data.go lines 318-323 to trace how the ingest_pipeline metricset selects its target in x-pack mode. Then inspect integrations/packages/elasticsearch/kibana/dashboard/elasticsearch-metrics-ingest-pipelines.json; done means the metricset data lands in the .monitoring-* data stream and the dashboard reflects that target.

Written by the indexing model from the issue text.

Assessment

Tech stack
elasticsearch, go
Domain
backend, observability-sre
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
52/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.