Graylog2 / Graylog2/graylog2-server
Expose search latency histograms
- Dominant language
- Java
- Stars
- 8.1k
- Forks
- 1.1k
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 217
Description
## What?
The time it takes to service a search request is an important metric for measuring system health and user experience.
Due to the very flexible nature of search queries, their potential high volume, and the mechanics of their execution, it's not trivial to measure a single value to reflect the health of a Graylog cluster. Thus we need a slightly more sophisticated system.
The most obvious way to measure it is to record the elapsed time and result set size for every query in the system, but the skewed distributions of different time ranges and query complexities will render the result relatively useless.
Another option is to pick time ranges of interest and use them to build up histograms for each bucket (this could follow the default search time presets we ship with). To make sense over time, the histograms should decay relatively quickly. Otherwise, it would smooth out too much over time.
Recording and processing the raw events for each search would yield the best results, but that requires extra processing, is much more difficult to scale, and is more complex to query as well.
## Why?
The time it takes to execute the default search has a very high impact on the perceived performance of the system. Understanding how this latency behaves and develops over time can be very helpful in tuning and troubleshooting a system.
Since the work necessary for running more complex queries over larger time frames and datasets is significantly different than the default `*`, two-widget, 5-minute search window, we can't just collect the minimum, maximum, standard deviation, mean, and the number of search requests. The difference in distribution is too large to make any sense of it, especially over larger amounts of time.
## Your Environment
* Graylog Version: 5.1
* OpenSearch Version:
* MongoDB Version:
* Operating System:
* Browser version:
Contributor guide
Assessment
This issue has not been assessed yet.