Refactor benchmarking code
- Dominant language
- Python
- Stars
- 43
- Forks
- 6
- Avg merge
- 3d 22h
- Merged PRs (30d)
- 10
Description
### What's the new feature?
As the number of solvers has grown, updating benchmarking results has become cumbersome. Ideally, we would have single script to define the solvers and run the set of benchmarks.
Some identified issues:
- Currently most solvers are setup in `coreax/benchmark_util.py` in the `initialise_solvers()` function. However `benchmark/blobs_benchmark.py` has separate code for this, which requires defining new methods in both places.
- Make the save paths more robust to make sure benchmarking results and images are saved to the appropriate location (currently `examples/benchmarking_images`) regardless of where the script is run from.
- Consider creating a script that contains basic configurations, runs all benchmarking and visualisation scripts, and updates the `benchmark.rst` documentation page.
- Document the benchmarking process and conventions in CONTRIBUTING.md.
- Refactor existing code to avoid local imports from other coreax packages.
- Make sure the code adheres to the usual standards, e.g., for docstrings and typing.
- Investigate optimisation possibilities, e.g., caching some parts of computation, parallelising test runs.
Some suggestions are given in #1000.
### What value does this add?
Easier to benchmark new and existing coreset algorithms.
### Is there an alternative you've considered?
_No response_
### Additional context
_No response_
Contributor guide
Research direction
Start by reading coreax/benchmark_util.py, especially initialise_solvers(), and benchmark/blobs_benchmark.py, then review the suggestions in #1000. Trace how benchmark outputs and images are currently saved, and inspect benchmark.rst and CONTRIBUTING.md. Done means benchmarking has a single maintainable configuration and execution path, robust output locations, documented conventions, and consistent code standards.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- documentation, performance, tooling
- Issue type
- Refactor
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100