uber / uber/h3

Performance benchmarking improvements

Open
#504 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
C
Stars
6.5k
Forks
627
Avg merge
3d 21h
Merged PRs (30d)
6

Description

As discussed offline, we think there are a few improvements we could make in how we do performance benchmarks for H3:

  • use some standard benchmarking framework instead of rolling our own (which is what we currently do)
    • a framework could provide nice features like warm-ups, advanced stats (min vs avg), etc.
    • what options currently exist for C?
  • standardize the benchmark format to make it more machine-readable (like in Go), which would make it easier to automatically compare timings between diffs
  • avoid doing benchmarks on laptops and/or finding a dedicated benchmarking machine
  • random (but deterministic) inputs for improved input coverage
  • benchmark performance tracking/plotting over the timeline of diffs, like what is done in Python using asv for libraries like numpy

For context, this issue came up after seeing a lot of variance in the benchmarks I was running on my laptop for https://github.com/uber/h3/pull/496

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by locating H3's current performance benchmark implementation, since the issue does not name files or tests. Review C benchmarking frameworks and the linked Go benchmark format, then compare the possible scope with Python's asv approach. Done would require an agreed design for benchmark execution, machine-readable results, deterministic inputs, and performance tracking.

Written by the indexing model from the issue text.

Assessment

Tech stack
c
Domain
performance, testing
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.