We should support outputting quantile results
Open
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 1k
- Forks
- 346
- PR merge metrics
- No merged PRs in 30d
Description
Since one operator can have multiple inputs, the average results can be misleading. When the optimized operator is faster than the baseline operator on average, but some quantiles are slower, we might overlook performance inefficiencies. We should report the quantile results for each operator.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
The issue names no files, tests, or entry points. Start by tracing where benchmark results are aggregated and displayed, then determine how quantiles should be calculated per operator; done means reports show quantile results alongside averages for each operator.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, pytorch
- Domain
- performance
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 42/100