JuliaPy / JuliaPy/PythonCall.jl
Formalise Benchmarks
- Langage dominant
- Julia
- Étoiles
- 1.1k
- Forks
- 86
- Merge moyen
- 1 j 22 h
- PR mergées (30 j)
- 3
Description
## Rationale
Create a formal benchmark pipeline to compare
- Python
- PythonCall (dev)
- PythonCall (stable)
- PyCall
_Originally posted by @cjdoris in https://github.com/cjdoris/PythonCall.jl/issues/300#issuecomment-1547350528_
### Requirements
1. **Match benchmark cases across suites**
2. **Use the same Python executable across all interfaces**
3. **Store multiple results or condensed statistics**
4. **Track memory usage**
### Comments
#### Julia Side
Most benchmarking tools in Julia run atop **BenchmarkTools.jl[^BenchmarkTools.jl]** and using their interface to define test suites and store results is the way to go. Both **PkgBenchmark.jl[^PkgBenchmark.jl]** and **AirspeedVelocity.jl[^AirspeedVelocity.jl]** provide functionality to compare multiple versions of a *single* package. Yet, they don't support comparison across *multiple* packages out-of-the-box. There will be some homework for us in building the right tools for this slightly generalized toolset.
Important to say that **PkgBenchmark.jl** has useful methods in its public API that we could leverage to build what we need. This includes methods for comparison between suites and for exporting those results to Markdown. **AirspeedVelocity.jl** is only made available through the CLI.
#### Python Side
In order to enjoy the same level of detail providede by **BenchmarkTools.jl**, we should adopt **pyperf[^pyperf]**.
There are many ways to use it, but a few experiments showed that the CLI + JSON interface is probably the desired option.
For each test case, stored in the `PY_CODE` variable, we would then create a temporary path `JSON_PATH` and run
```julia
run(`$(PY_EXE) -m pyperf timeit "$(PY_CODE)" --append="$(JSON_PATH)" --tracemalloc`)
```
After that, we should be able parse the [output JSON](https://pyperf.readthedocs.io/en/latest/api.html#pyperf-json-format) and convert it into a [`PkgBenchmark.BenchmarkResults`](https://juliaci.github.io/PkgBenchmark.jl/dev/run_benchmarks/#PkgBenchmark.BenchmarkResults) object. This makes it easier for integrating those results in the overall machinery, reducing the problem to setting the `Python` result as the reference value.
## Tasks
- [ ] Implement the reference Python benchmark cases
- [ ] Implement the corresponding versions in the other suites
- [ ] PythonCall (dev)
- [ ] PythonCall (stable)
- [ ] PyCall
- [ ] Write a translator for **pyperf JSON** into `BenchmarkResults`
- [ ] Write comparison tools
- [ ] Write report generator
- [ ] Setup GitHub actions
## Resources
- [BENCHMARKS.md](https://github.com/cjdoris/PythonCall.jl/blob/main/BENCHMARKS.md)
## References
[^BenchmarkTools.jl]:
[BenchmarkTools.jl](https://juliaci.github.io/BenchmarkTools.jl/stable/)
[^pyperf]:
[pyperf](https://pyperf.readthedocs.io/en/latest/)
[^AirspeedVelocity.jl]:
[AirspeedVelocity.jl](https://github.com/MilesCranmer/AirspeedVelocity.jl)
[^PkgBenchmark.jl]:
[PkgBenchmark.jl](https://github.com/JuliaCI/PkgBenchmark.jl/)
Guide de contribution
Aucun guide de contribution indexé pour ce dépôt
Piste de recherche
Start with BENCHMARKS.md, the pyperf JSON format, and the referenced BenchmarkTools.jl and PkgBenchmark.jl APIs. The work is complete when matching cases, shared Python executables, stored statistics and memory usage are supported across the listed suites, with comparison, reporting, and GitHub Actions tasks addressed.
Rédigé par le modèle d'indexation à partir du texte de l'issue.
Évaluation
- Stack technique
- julia, python
- Domaine
- ci-cd, performance, testing, tooling
- Type d'issue
- Fonctionnalité
- Difficulté
- 5/5
- Temps estimé
- Plus d'une semaine
- Activité
- À l'abandon
- Clarté
- À clarifier
- Accessibilité débutants
- 25/100