[Metricbeat] Improve the `elasticsearch` module when used for Stack Monitoring
- Dominant language
- Go
- Stars
- 12.7k
- Forks
- 5k
- Avg merge
- 2d 15m
- Merged PRs (30d)
- 385
Description
While investigating the root cause of [indexing failures](https://github.com/elastic/elasticsearch/issues/107601) (also reported [here](https://github.com/elastic/elasticsearch/issues/80802#issuecomment-1285039063) in the past), we discovered that when using Metricbeat to feed Stack Monitoring, the `elasticsearch` module of Metricbeat ships `elasticsearch.shard` documents with concrete IDs that are made of the [current cluster state](https://github.com/elastic/beats/blob/main/metricbeat/module/elasticsearch/shard/data.go#L164-L167) (i.e., `state_uuid`) and some other constant data. Since the cluster state doesn't change at the same pace as Metricbeat collection rounds (10s by default), those version conflicts happen all the time.
Those version conflicts are probably a side-effect of switching to data streams in 8.0.0 (i.e. put if absent semantics with concrete ID) and weren't apparent earlier when the data was stored in simple indexes. Since each `elasticsearch.shard` document is about a shard placement in the cluster, the logic makes sense, i.e. there's no point re-indexing a document whose content hasn't changed since the last collection round.
However, we could/should go one step further and detect if the cluster state hasn't changed between two collection rounds. I'm naively thinking about "simply" comparing the old and new `state_uuid`, but it might be more involved than that. Anyway, if there's no change, there's no point in even rebuilding those documents and sending them again, since we know they'll bounce anyway, generate a version conflict and increase the indexing failure counter for no reason. In addition to that, that wastes network bandwidth and CPU/RAM resource on ES side. For big clusters with many thousands of shards, that can make a big difference.
Related issue: https://github.com/elastic/beats/issues/36547#issuecomment-2064306586
Contributor guide
Research direction
Start in metricbeat/module/elasticsearch/shard/data.go, especially the code that builds shard documents from the current cluster state and state_uuid. Trace how collection rounds send these documents to Stack Monitoring and determine how unchanged state_uuid values should be handled. Done means unchanged shard placements are not rebuilt or sent, while changed cluster state still produces the required documents without version-conflict retries.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- elasticsearch, go
- Domain
- observability-sre
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100