marigold-dev / marigold-dev/deku
Standardize benchmarking output
Nobody has claimed this yet.
- Dominant language
- OCaml
- Stars
- 82
- Forks
- 17
- PR merge metrics
- No merged PRs in 30d
Description
Our benchmarks should be organized so that they produce a uniform, machine-readable output. Here's what I have in mind:
Make a module interface BENCHMARK to which all the benchmarks will confirm. Then create a function that iterates over a list of such modules, runs them, and outputs the logs, or optionally outputs a csv of the results.
I imagine that to keep things simple benchmarks will have only a few dimensions:
description, domains, load size, runtime, ops/second
where
- domains is set once from an environment variable
- ops/second is derived from the load size and the runtime
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
No files, tests, or entry points are named. First locate the existing benchmark implementations and their current output path, then determine how the proposed BENCHMARK interface and runner should fit them. Done means all benchmarks produce uniform logs and optionally machine-readable CSV results with the listed dimensions.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- ocaml
- Domain
- performance
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100