gchq / gchq/coreax

Refactor benchmarking code

Open
#1,002 1 comment 0 reactions 0 assignees View on GitHub
enhancement help wanted
Dominant language
Python
Stars
43
Forks
6
Avg merge
3d 22h
Merged PRs (30d)
10

Description

### What's the new feature?

As the number of solvers has grown, updating benchmarking results has become cumbersome. Ideally, we would have single script to define the solvers and run the set of benchmarks.

Some identified issues:

- Currently most solvers are setup in `coreax/benchmark_util.py` in the `initialise_solvers()` function. However `benchmark/blobs_benchmark.py` has separate code for this, which requires defining new methods in both places.
- Make the save paths more robust to make sure benchmarking results and images are saved to the appropriate location (currently `examples/benchmarking_images`) regardless of where the script is run from.
- Consider creating a script that contains basic configurations, runs all benchmarking and visualisation scripts, and updates the `benchmark.rst` documentation page.
- Document the benchmarking process and conventions in CONTRIBUTING.md.
- Refactor existing code to avoid local imports from other coreax packages.
- Make sure the code adheres to the usual standards, e.g., for docstrings and typing.
- Investigate optimisation possibilities, e.g., caching some parts of computation, parallelising test runs.

Some suggestions are given in #1000.

### What value does this add?

Easier to benchmark new and existing coreset algorithms.

### Is there an alternative you've considered?

_No response_

### Additional context

_No response_

Contributor guide

Open the contributing guide

Research direction

Start by reading coreax/benchmark_util.py, especially initialise_solvers(), and benchmark/blobs_benchmark.py, then review the suggestions in #1000. Trace how benchmark outputs and images are currently saved, and inspect benchmark.rst and CONTRIBUTING.md. Done means benchmarking has a single maintainable configuration and execution path, robust output locations, documented conventions, and consistent code standards.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
documentation, performance, tooling
Issue type
Refactor
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.