RFC: Add `report` subcommand
- Dominant language
- Python
- Stars
- 2k
- Forks
- 347
- Avg merge
- 6d 17h
- Merged PRs (30d)
- 5
Description
## Summary
Today, users often run multiple races in order to have confidence in the results. They then either eyeball N summary reports, or use Elasticsearch aggregations against the raw samples in `rally-metrics-*` to get a better picture of the results and/or better understand the variability.
There is also no way to re-generate the summary report for a race after the fact. The report is printed once, at the end of a `race`.
Therefore, Rally should offer:
1. An easy way to re-generate the summary report for a single, existing race using the Rally CLI
2. An easy way to calculate statistics across N races using the Rally CLI
3. An easy way to report the variability/variance of tasks across those races using the Rally CLI
I propose the following:
1. An `esrally report` CLI subcommand that re-generates the summary report for one existing race.
2. When multiple race-ids are passed, `esrally report` generates a single summary by applying statistical aggregations across the samples of all the given races.
3. Reporting between-race variance through a [Coefficient of Variation (CV)](https://en.wikipedia.org/wiki/Coefficient_of_variation) column in the multi-race `esrally report` output.
## 1. Re-generate the report for a single race
`esrally report --race=` prints the summary report for an existing race, the same table you see at the end of `esrally race`.
By default the numbers come from the stored results (`rally-results-*`, or `race.json` for a file-based store), so this matches the original report exactly, is fast, and works even if the raw samples have since been deleted by ILM. If stored results are unavailable, Rally falls back to recomputing them from the raw samples in `rally-metrics-*` (the same path used for the multi-race case below).
### CLI
```
esrally report --race=
```
### Behaviour
* Resolves the race the same way `compare` resolves `--baseline` / `--contender`. A missing race ID is an error.
* Prefers stored results, but falls back to recomputing from raw samples in `rally-metrics-*` when stored results are missing.
* Honours `--report-format` and `--report-file` (see options below).
* Read-only: it does not write new documents to the metrics store.
* No `CV` column is shown for a single race.
## 2. Aggregate results over N races
When more than one race is selected, `esrally report` prints one summary report computed across the samples of all the selected races, per task. The races must share a track and challenge.
### CLI
```
# explicit race IDs
esrally report --race=,,
# or select races the same way as `list races`
esrally report --track= [--challenge=] [--user-tags=] [--limit=N]
```
`--race` selects specific race-ids. If it is not set, races are selected with the same arguments as `list races` (`--track`, `--challenge`, `--user-tags`, `--benchmark-name`, `--from-date`, `--to-date`, `--limit`). If `--race` is set, the filter flags are ignored.
### Options
| Option | Behaviour |
| --- | --- |
| `--race ` | One race ID, or a comma-separated list. If set, the filter flags are ignored. |
| `--track` / `--challenge` / `--user-tags` / `--benchmark-name` / `--from-date` / `--to-date` / `--limit` | Filter mode, same meaning as `esrally list races`. |
| `--report-format` markdown / csv | Same as `compare`. |
| `--report-file` | Also write the table to a file. |
### Behaviour
* Multi-race aggregation requires `datastore.type = elasticsearch`, as raw samples are not kept after an in-memory race.
* It requires the selected races to share the same track and challenge. A missing race ID is an error. Differing `car` or `distribution-version` is a warning, not an error.
* Per-task request metrics (`throughput`, `latency`, `service_time`, `processing_time`) are computed from the pooled raw samples in `rally-metrics-*` (e.g. `terms` on `race-id`, `sample-type: normal` only, `extended_stats` / `percentiles`). Warmup samples are excluded.
* Cluster-level, one-shot metrics (e.g. `Cumulative indexing time`, `Total Young Gen GC time`, `Dataset size`) are single values per race rather than sample streams. Their `Aggregate` value is the mean of the per-race values, and `CV` is computed over those same per-race values.
* Tasks are collected from the `op_metrics` of the selected races.
* Read-only: it does not write new documents to the metrics store.
## 3. Variance (CV)
For each row shown in the multi-race table, `esrally report` also reports the between-race [Coefficient of Variation (CV)](https://en.wikipedia.org/wiki/Coefficient_of_variation). The CV summarises the repeatability of that result across the N races.
The `Aggregate` column and the `CV` column are computed differently, and this distinction matters:
* `Aggregate` is computed from the pooled `normal` samples of all selected races (for one-shot cluster metrics, the mean of the per-race values).
* `CV` shows how much a metric changed from race to race. Take that metric's value from each race (say the mean throughput from all 10 races), and `CV` is their standard deviation divided by their average, written as a percentage. A low `CV` (e.g. ~1-2%) means the races closely agreed. A high `CV` (e.g. ~10-20%) means the result swung a lot between runs. Note the average used here is the plain average of the 10 per-race values, which can differ slightly from the `Aggregate` column (that one is calculated from every raw sample in `rally-metrics*`).
CV is shown as `-` when it cannot be computed, e.g. fewer than two per-race values, or a mean of zero (e.g. an all-zero error rate).
| Metric | Task | Aggregate | CV | Unit |
| ---------------------------- | ------------ | --------: | ----: | ------ |
| Min Throughput | index-append | 19118 | 6.1% | docs/s |
| Mean Throughput | index-append | 20232 | 3.4% | docs/s |
| Median Throughput | index-append | 20110 | 3.1% | docs/s |
| Max Throughput | index-append | 21172 | 5.8% | docs/s |
| 50th percentile latency | index-append | 803.42 | 4.0% | ms |
| 90th percentile latency | index-append | 1913.70 | 7.2% | ms |
| 99th percentile latency | index-append | 3591.23 | 12.4% | ms |
| 100th percentile latency | index-append | 6642.97 | 18.1% | ms |
| 50th percentile service time | index-append | 612.10 | 3.8% | ms |
| 90th percentile service time | index-append | 1402.44 | 6.9% | ms |
| 99th percentile service time | index-append | 2810.02 | 11.8% | ms |
| error rate | index-append | 0.00 | - | % |
| Mean Throughput | term | 50.12 | 2.1% | ops/s |
| 50th percentile latency | term | 2.13 | 5.5% | ms |
| 90th percentile latency | term | 2.81 | 8.0% | ms |
| 99th percentile latency | term | 4.72 | 14.6% | ms |
| 100th percentile latency | term | 5.15 | 22.0% | ms |
| error rate | term | 0.00 | - | % |
| Cumulative indexing time | | 55.80 | 2.8% | min |
| Total Young Gen GC time | | 4.12 | 9.5% | s |
| Dataset size | | 3.05 | 0.4% | GB |
## Out of scope
* Persisting an aggregated race as a new race / results document.
* A `CV` column on `esrally compare`.
* Comparing two *groups* of races against each other (each side aggregated, then diffed).
Contributor guide
Research direction
Start by tracing the existing `compare` and `list races` CLI entry points and their handling of `rally-results-*`, `race.json`, and `rally-metrics-*`. Define the `esrally report` behavior for single and multi-race selection, pooled metrics, CV output, and read-only operation; done means the specified CLI options and aggregation rules work without writing new metrics.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- elasticsearch, python
- Domain
- cli, data, databases
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100