Compare baseline results with more than one contender
- Dominant language
- Python
- Stars
- 2k
- Forks
- 347
- Avg merge
- 6d 17h
- Merged PRs (30d)
- 5
Description
**Describe the feature**:
Rally to be able to accept more than one contender when doing baseline comparison and present "diff" results in another column side by side.
Currently Rally only support one contender when doing baseline comparison:
```
Recent races:
Race ID Race Timestamp Track Track Parameters Challenge Car User Tags Track Revision Team Revision
------------------------------------ ---------------- ------- ------------------ ------------------- -------- --------------------------- ---------------- ---------------
59913edf-8b24-4023-bac4-9b5bb4a20109 20200317T112937Z pmc append-no-conflicts external pmc=sensor_agressive_policy 72094f3
9e44f580-6c89-4047-9904-effabbff55b1 20200317T102335Z pmc append-no-conflicts external scenario=without_sensor 72094f3
4c2bbfca-5145-4ecd-bc33-4c4b19edf257 20200317T074652Z pmc append-no-conflicts external 72094f3
-------------------------------
[INFO] SUCCESS (took 2 seconds)
-------------------------------
$ esrally compare --baseline=9e44f580-6c89-4047-9904-effabbff55b1 --contender=59913edf-8b24-4023-bac4-9b5bb4a20109 --contender=4c2bbfca-5145-4ecd-bc33-4c4b19edf257
____ ____
/ __ \____ _/ / /_ __
/ /_/ / __ `/ / / / / /
/ _, _/ /_/ / / / /_/ /
/_/ |_|\__,_/_/_/\__, /
/____/
Comparing baseline
Race ID: 9e44f580-6c89-4047-9904-effabbff55b1
Race timestamp: 2020-03-17 10:23:35
Challenge: append-no-conflicts
Car: external
User tags: scenario=without_sensor
with contender
Race ID: 4c2bbfca-5145-4ecd-bc33-4c4b19edf257
Race timestamp: 2020-03-17 07:46:52
Challenge: append-no-conflicts
Car: external
```
Would be good if we can add more than one contender and present results after comparison side by side.
Contributor guide
Research direction
Start by tracing the `esrally compare` command and its current single `--contender` handling. Confirm how comparison results are assembled and identify the relevant tests before defining the multi-contender behavior. Done means multiple contenders are accepted and their diff results are presented side by side.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- cli
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100