[Meta][Metricbeat] - Collect even more additional Elasticsearch node metrics for enhanced dashboards
- Dominant language
- Go
- Stars
- 12.7k
- Forks
- 5k
- Avg merge
- 2d 15m
- Merged PRs (30d)
- 385
Description
In order to improve stack monitoring, there needs to be some additional datapoints made available in a flattened view to enable visualizations and alerting.
Some of this data exists in the cluster_stats, but it being nested makes it very difficult to utilize.
Data that we would like to have either in the node_stats metricset (or a new metricset) is largely available by calling the _nodes API. Period for this might make sense to be different from existing node_stats as these items would not change often.
From the _nodes API
node.roles - this being stored in arrays, causes a single node to appear as multiple rows in a table visualization (one for each role), making it difficult to get a concise, single-line overview of a node. could this be as a single list instead?
node.attributes.availability_zone
node.attributes.region
node.instance_configuration - cloud only, but useful if available
node.version (i think this would be useful to track status of rolling updates)
node.os.cpu.processors: The number of logical CPU cores. This provides the denominator for calculating CPU load percentage, allowing for more effective alerting thresholds (e.g., load_avg_1m / processors > 0.8).
Thread Pool Statistics: Exposing the size and queue size for key thread pools is vital for diagnosing bottlenecks.
node.thread_pool.force_merge.size
node.thread_pool.force_merge.queue_size
node.thread_pool.get.size
node.thread_pool.get.queue_size
node.thread_pool.search.size
node.thread_pool.search.queue_size
node.thread_pool.write.size
node.thread_pool.write.queue_size
node.thread_pool.management.size
node.thread_pool.management.queue_size
node.thread_pool.esql_worker.size
node.thread_pool.esql_worker.queue_size
node.thread_pool.refresh.size
node.thread_pool.refresh.queue_size
node.thread_pool.snapshot.size
node.thread_pool.snapshot.queue_size
Quality-of-Life Improvement: Pre-calculated Percentages
As an additional, related enhancement, providing pre-calculated disk and memory usage percentages would be highly beneficial. While the raw used_in_bytes and total_in_bytes fields are available, exporting a calculated disk.used.pct and memory.used.pct field would simplify dashboard creation and alerting for all users.
Contributor guide
Research direction
Start by inspecting the Metricbeat node_stats metricset and existing cluster_stats data, then compare the available fields with the Elasticsearch _nodes API. Define which node roles, attributes, version, processor, thread-pool, disk, and memory values belong in the metricset, and verify that the resulting fields are flattened and suitable for dashboards and alerting.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- observability-sre
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 42/100