RustCrypto / RustCrypto/KEMs

Use Codspeed for testing

Open
#24 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Rust
Stars
111
Forks
53
Avg merge
5d 29m
Merged PRs (30d)
2

Description

Codspeed is a performance testing tool designed to produce highly consistent performance measurements. It also integrates very cleanly with criterion.rs.

I prototyped this out in https://github.com/bifurcation/KEMs/pull/1. (Unfortunately, that also includes a bunch of noise due to main branch divergence. The main commit is https://github.com/bifurcation/KEMs/pull/1/commits/30a1c599b3e8fb114165e475eafbfde8e7922575.) In addition to those code changes, an admin for the RustCrypto org would have to enable Codspeed for the repo and provision the required secret.

I'm having a little trouble having confidence in the measurements, though. Looking at https://github.com/bifurcation/KEMs/pull/3 and its Codspeed report, it is measurably slower on my local machine, but Codspeed reports marginal performance improvements. The execution times are also off by about an order of magnitude.

keygen                  time:   [37.873 µs 37.899 µs 37.935 µs]
                        change: [+1.6945% +2.8309% +3.4566%] (p = 0.00 < 0.05)
                        Performance has regressed.

encapsulate             time:   [36.971 µs 37.129 µs 37.401 µs]
                        change: [+1.4643% +1.7390% +2.1480%] (p = 0.00 < 0.05)
                        Performance has regressed.

decapsulate             time:   [45.140 µs 45.368 µs 45.625 µs]
                        change: [+0.5023% +0.9468% +1.4248%] (p = 0.00 < 0.05)
                        Change within noise threshold.

round_trip              time:   [109.86 µs 110.04 µs 110.27 µs]
                        change: [+1.3724% +1.8421% +2.2354%] (p = 0.00 < 0.05)
                        Performance has regressed.

The version in that prototype is pretty slow, because it builds everything afresh, as @newpavlov noted in the accidental PR #20. If we're going to do this, we should do something like @newpavlov suggests to make it faster.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Review the prototype in bifurcation/KEMs pull request #1, especially commit 30a1c599b3e8fb114165e475eafbfde8e7922575, then compare its measurements with pull request #3 and the linked Codspeed report. Check the benchmark setup using criterion.rs and the faster approach suggested in accidental pull request #20. Done means the repository uses Codspeed with the required RustCrypto configuration and produces trusted, efficient performance measurements.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
ci-cd, performance, testing-qa
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.