cucapra / cucapra/pollen

Systematic performance measurement

Open
#229 0 comments 0 reactions 1 assignee Claimed by @johnpalsberg View on GitHub
Dominant language
Rust
Stars
45
Forks
3
PR merge metrics
No merged PRs in 30d

Description

We have been doing a lot of _ad hoc_ measurement of the performance and file size of FlatGFA, but we need this to be more systematic. Here is a list of desiderata:

* Compare FlatGFA's running time and file size against other tools, including odgi.
* For performance, measure the time it takes to convert files (from GFA to both `.og` and `.flatgfa` formats) and for several analysis operations (e.g., `fgfa depth` vs. `odgi depth`). Make it easy to add comparisons for new commands.
* Use a collection of medium- to large-sized GFA files as input. Make it easy to add new files.
* Automatically run the benchmark suite on every commit to `main` and on every PR. Possibly use an existing commercial product, such as [Codspeed](https://codspeed.io) or [Bencher](https://bencher.dev) for this.
* Also provide a convenient way to run the benchmarks manually & locally (i.e., it shouldn't be _necessary_ to push to `main` to run the benchmark suite).

I have started a very rough sketch of what this might look like [in `bench.py` in this repository](https://github.com/cucapra/pollen/blob/e0cf9284b411fe201e8168b8e4d978765e1cfc9c/bench/bench.py). This does not yet do close to everything we need, and we may need to throw it away and start over, but there are a _few_ things that I like about the current design. First, there is a configuration file for declaratively specifying which experiments to run, like this:
https://github.com/cucapra/pollen/blob/e0cf9284b411fe201e8168b8e4d978765e1cfc9c/bench/config.toml#L1-L20

The tool can also automatically download sets of graphs from public repositories, as configured here:
https://github.com/cucapra/pollen/blob/e0cf9284b411fe201e8168b8e4d978765e1cfc9c/bench/graphs.toml#L1-L4

That seems kind of nice. I am also happy with my little Python wrapper for [hyperfine](https://github.com/sharkdp/hyperfine) invocations:
https://github.com/cucapra/pollen/blob/e0cf9284b411fe201e8168b8e4d978765e1cfc9c/bench/bench.py#L68

It's a simple, reasonably safe interface to the underlying tool executions.

To make the evaluation harness of our dreams, I suggest we proceed with these steps:

* [ ] Make a truly MVP measurement tool that can only measure one thing: for example, GFA conversion time.
* [ ] Try to set up CI for this tool, hopefully by integrating with Bencher or Codspeed. For example, Bencher seems to support [custom benchmark results](https://bencher.dev/docs/how-to/track-custom-benchmarks/); we could report our results in this format or just use its existing harness off the shelf. Bencher even seems to have a mode for [reporting file sizes](https://bencher.dev/docs/how-to/track-file-size/), which is relevant to us.
* [ ] Expand our coverage of both graphs and measured commands.
* [ ] Write documentation for other team members, so they can easily add their command comparisons to the benchmarking workflow.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.