Output the probability of being faster (slower) when compare results
Open
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 962
- Forks
- 106
- Avg merge
- 2d 18h
- Merged PRs (30d)
- 4
Description
When compare two results it would be helpful to output a probability of one result be faster then other.
If times1 and times2 are sets of measured times, then the probability of the first benchmark being faster than the second one is estimated as:
sum(x < y for x in times1 for y in times2)/len(times1)/len(times2)
Actually you can sort one of sets and use binary search for optimization.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by locating the code that compares two benchmark results and the tests covering comparison output. Implement the probability calculation described in the issue, including the faster and slower cases, and verify that comparison output reports the resulting probability.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- performance
- Issue type
- Feature
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100