Improve dev workflows around benchmarks
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 394
- Forks
- 95
- Avg merge
- 4d 3h
- Merged PRs (30d)
- 7
Description
The current workflow for identifying regressions and benchmarking packages is not ideal.
Some comments/ideas for improvement:
spatialdataandnapari-spatialdataboth utilizeasv, but in different ways: in particular,spatialdatarequires manual runs, whilenapari-spatialdatahas workflows enabled.- The benchmarking machinery in
napari-spatialdatais complex and should be simplified. - We lack a benchmarks results page, making the exploration of benchmark results cumbersome: https://pv.github.io/numpy-bench/#?sort=3&dir=desc
- It appears that with the current workflow, we cannot easily benchmark versions with incompatible installs, which are precisely the scenarios where benchmarks would be most useful: https://github.com/scverse/napari-spatialdata/actions/runs/20647610286/job/59287297836?pr=377
Path forward: Currently, the focus on spatialdata is at the specs and APIs level, not performance. Therefore, I would consciously place less emphasis on systematic benchmarks for the time being and also disable the benchmarks workflow in napari-spatialdata (which can still be used on-demand, similar to spatialdata). When we shift our focus to performance, we should revamp the benchmarks suite, enable it universally (not just in napari-spatialdata), document it for easier onboard for new devs, and enhance it for systematic use.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by comparing how spatialdata and napari-spatialdata use asv, including the linked GitHub Actions run and the existing benchmark workflow. The issue names no files or tests; clarify the desired scope and agree whether completion means disabling the current workflow or designing the broader benchmark, results-page, compatibility, and onboarding plan.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- github-actions, python
- Domain
- ci-cd, performance
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100