elastic / elastic/rally

Compare baseline results with more than one contender

Open
#933 0 comments 0 reactions 0 assignees View on GitHub
:Reporting enhancement help wanted
Dominant language
Python
Stars
2k
Forks
347
Avg merge
6d 17h
Merged PRs (30d)
5

Description

**Describe the feature**:
Rally to be able to accept more than one contender when doing baseline comparison and present "diff" results in another column side by side.

Currently Rally only support one contender when doing baseline comparison:

```
Recent races:
Race ID Race Timestamp Track Track Parameters Challenge Car User Tags Track Revision Team Revision
------------------------------------ ---------------- ------- ------------------ ------------------- -------- --------------------------- ---------------- ---------------
59913edf-8b24-4023-bac4-9b5bb4a20109 20200317T112937Z pmc append-no-conflicts external pmc=sensor_agressive_policy 72094f3
9e44f580-6c89-4047-9904-effabbff55b1 20200317T102335Z pmc append-no-conflicts external scenario=without_sensor 72094f3
4c2bbfca-5145-4ecd-bc33-4c4b19edf257 20200317T074652Z pmc append-no-conflicts external 72094f3
-------------------------------
[INFO] SUCCESS (took 2 seconds)
-------------------------------
$ esrally compare --baseline=9e44f580-6c89-4047-9904-effabbff55b1 --contender=59913edf-8b24-4023-bac4-9b5bb4a20109 --contender=4c2bbfca-5145-4ecd-bc33-4c4b19edf257
____ ____
/ __ \____ _/ / /_ __
/ /_/ / __ `/ / / / / /
/ _, _/ /_/ / / / /_/ /
/_/ |_|\__,_/_/_/\__, /
/____/
Comparing baseline
Race ID: 9e44f580-6c89-4047-9904-effabbff55b1
Race timestamp: 2020-03-17 10:23:35
Challenge: append-no-conflicts
Car: external
User tags: scenario=without_sensor
with contender
Race ID: 4c2bbfca-5145-4ecd-bc33-4c4b19edf257
Race timestamp: 2020-03-17 07:46:52
Challenge: append-no-conflicts
Car: external
```

Would be good if we can add more than one contender and present results after comparison side by side.

Contributor guide

Open the contributing guide

Research direction

Start by tracing the `esrally compare` command and its current single `--contender` handling. Confirm how comparison results are assembled and identify the relevant tests before defining the multi-contender behavior. Done means multiple contenders are accepted and their diff results are presented side by side.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
cli
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.