[Metricbeat][Elasticsearch][ingest_pipeline] In x-pack mode, data is sent to the wrong target
- Dominant language
- Go
- Stars
- 12.7k
- Forks
- 5k
- Avg merge
- 2d 15m
- Merged PRs (30d)
- 385
Description
When running in x-pack mode, it is expected that all data from the Elasticsearch module should land in the `.monitoring-*` data stream, but the `ingest_pipeline` metricset doesn't [override](https://github.com/elastic/beats/blob/main/metricbeat/module/elasticsearch/cluster_stats/data.go#L318-L323) the target data stream before the events are shipped, leading them to land in `metricbeat-*`.
Fixing this issue in the Beats code is trivial, but it will also require us to update the Elasticsearch Ingest Pipeline [dashboard](https://github.com/elastic/integrations/blob/main/packages/elasticsearch/kibana/dashboard/elasticsearch-metrics-ingest-pipelines.json) found in the Elasticsearch integration.
Contributor guide
Research direction
Start at metricbeat/module/elasticsearch/cluster_stats/data.go lines 318-323 to trace how the ingest_pipeline metricset selects its target in x-pack mode. Then inspect integrations/packages/elasticsearch/kibana/dashboard/elasticsearch-metrics-ingest-pipelines.json; done means the metricset data lands in the .monitoring-* data stream and the dashboard reflects that target.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- elasticsearch, go
- Domain
- backend, observability-sre
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 52/100