IntersectMBO / IntersectMBO/plutus
Improve output of benchmark tests
- Dominant language
- Haskell
- Stars
- 1.6k
- Forks
- 508
- Avg merge
- 3d 10h
- Merged PRs (30d)
- 22
Description
At least some of the tests in plutus-benchmark compare golden files contain the “abstract” cpu and mem usage. If these results change, then the developer must carefully compare the results to see if the abstract resource usage has increased. In reality, we should only consider a test as “failed” if the resource usage increases.
Comment from @zliu41 :
I don’t think we can make a test succeed if the cost decreases. Otherwise, in PR1 the cost may go down 10% and it goes unnoticed. Then in PR2 the cost may go up 10% and it goes unnoticed again, because the cost is no bigger than the original value.
But there are ways we can make it easier to tell which tests have increased or decreased costs:
- Use different colors for tests whose cost go up vs. go down
- Show percentage difference
- NumericUnderscores
Contributor guide
Assessment
This issue has not been assessed yet.