JuliaPy / JuliaPy/PythonCall.jl

Formalise Benchmarks

オープン
#308 コメント 7 件 リアクション 0 件 担当者 0 名 GitHub で見る
priority
主要言語
Julia
スター
1.1k
フォーク
86
平均マージ
1日 22時間
マージ済み PR(30日)
3

説明

## Rationale

Create a formal benchmark pipeline to compare
- Python
- PythonCall (dev)
- PythonCall (stable)
- PyCall

_Originally posted by @cjdoris in https://github.com/cjdoris/PythonCall.jl/issues/300#issuecomment-1547350528_

### Requirements

1. **Match benchmark cases across suites**
2. **Use the same Python executable across all interfaces**
3. **Store multiple results or condensed statistics**
4. **Track memory usage**

### Comments

#### Julia Side

Most benchmarking tools in Julia run atop **BenchmarkTools.jl[^BenchmarkTools.jl]** and using their interface to define test suites and store results is the way to go. Both **PkgBenchmark.jl[^PkgBenchmark.jl]** and **AirspeedVelocity.jl[^AirspeedVelocity.jl]** provide functionality to compare multiple versions of a *single* package. Yet, they don't support comparison across *multiple* packages out-of-the-box. There will be some homework for us in building the right tools for this slightly generalized toolset.

Important to say that **PkgBenchmark.jl** has useful methods in its public API that we could leverage to build what we need. This includes methods for comparison between suites and for exporting those results to Markdown. **AirspeedVelocity.jl** is only made available through the CLI.

#### Python Side

In order to enjoy the same level of detail providede by **BenchmarkTools.jl**, we should adopt **pyperf[^pyperf]**.
There are many ways to use it, but a few experiments showed that the CLI + JSON interface is probably the desired option.

For each test case, stored in the `PY_CODE` variable, we would then create a temporary path `JSON_PATH` and run

```julia
run(`$(PY_EXE) -m pyperf timeit "$(PY_CODE)" --append="$(JSON_PATH)" --tracemalloc`)
```

After that, we should be able parse the [output JSON](https://pyperf.readthedocs.io/en/latest/api.html#pyperf-json-format) and convert it into a [`PkgBenchmark.BenchmarkResults`](https://juliaci.github.io/PkgBenchmark.jl/dev/run_benchmarks/#PkgBenchmark.BenchmarkResults) object. This makes it easier for integrating those results in the overall machinery, reducing the problem to setting the `Python` result as the reference value.

## Tasks

- [ ] Implement the reference Python benchmark cases
- [ ] Implement the corresponding versions in the other suites
- [ ] PythonCall (dev)
- [ ] PythonCall (stable)
- [ ] PyCall
- [ ] Write a translator for **pyperf JSON** into `BenchmarkResults`
- [ ] Write comparison tools
- [ ] Write report generator
- [ ] Setup GitHub actions

## Resources

- [BENCHMARKS.md](https://github.com/cjdoris/PythonCall.jl/blob/main/BENCHMARKS.md)

## References

[^BenchmarkTools.jl]:
[BenchmarkTools.jl](https://juliaci.github.io/BenchmarkTools.jl/stable/)

[^pyperf]:
[pyperf](https://pyperf.readthedocs.io/en/latest/)

[^AirspeedVelocity.jl]:
[AirspeedVelocity.jl](https://github.com/MilesCranmer/AirspeedVelocity.jl)

[^PkgBenchmark.jl]:
[PkgBenchmark.jl](https://github.com/JuliaCI/PkgBenchmark.jl/)


コントリビューションガイド

このリポジトリのコントリビューションガイドは索引されていません

調査の方向性

Start with BENCHMARKS.md, the pyperf JSON format, and the referenced BenchmarkTools.jl and PkgBenchmark.jl APIs. The work is complete when matching cases, shared Python executables, stored statistics and memory usage are supported across the listed suites, with comparison, reporting, and GitHub Actions tasks addressed.

索引モデルが issue の本文から書いたものです。

評価

技術スタック
julia, python
領域
ci-cd, performance, testing, tooling
issue の種類
機能追加
難易度
5/5
見積もり時間
1週間以上
活発さ
停滞
明瞭さ
説明が足りない
初心者へのやさしさ
25/100

新しい issue をメールで受け取る

初心者向けの GitHub issue を短くまとめたダイジェスト。