argumentcomputer / argumentcomputer/ci-workflows

Laundry list about benchmarking

Aperta
#49 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub
Lingua principale
Nessun dato sulla lingua
Stelle
0
Fork
3
Merge medio
59m
PR unite (30g)
2

Descrizione

## TL;DR

It would be awesome for bencher if we had two jobs `gpu-bench-comparative` and `bench-comparative` with the following properties:
- activate `asm` on x86_64,
- use or do not use `cuda` depending on the nature of the called job,
- they use `BENCH_OUTPUT=commit-comment` by default, because
- they both use `message-format=json` and pipe to a file named after bench & commit (as gpu-bench-ci does)

## In detail

I've been running 6-7 benches (comparative, so 2 runs per) every day, here's the pet peeves outcome:
- if I want to pipe the output (json OR criterion baseline[^1]) into a tool (resp. critcmp or criterion-table), I have to remember to pass an env variable `XXX_BENCH_OUTPUT = "commit-comment"`,
- one thing we've solved : the name of the var is per-repo,
- the other thing: I have to edit the justfile super often : mostly I run 3 benches in parallel, but I don't have 3 gpu-equipped machines
- there are niceties in the gpu-bench-ci job that are not in the bench job,

Arecibo example:
```
# Run CPU benchmarks
bench +benches:
#!/bin/sh
for bench in {{benches}}; do
cargo criterion --bench $bench
done

# Run CUDA benchmarks on GPU
gpu-bench +benches:
#!/bin/sh
# The `compute`/`sm` number corresponds to the Nvidia GPU architecture
# In this case, the self-hosted machine uses the Ampere architecture, but we want this to be configurable
# See https://arnon.dk/matching-sm-architectures-arch-and-gencode-for-various-nvidia-cards/
export CUDA_ARCH=$(nvidia-smi --query-gpu=compute_cap --format=csv,noheader | sed 's/\.//g')
export EC_GPU_CUDA_NVCC_ARGS="--fatbin --gpu-architecture=sm_$CUDA_ARCH --generate-code=arch=compute_$CUDA_ARCH,code=sm_$CUDA_ARCH"
export EC_GPU_FRAMEWORK="cuda"

for bench in {{benches}}; do
cargo criterion --bench $bench --features "cuda"
done

# Run CUDA benchmarks on GPU, tuned for CI on Linux x86_64
gpu-bench-ci +benches:
#!/bin/sh
printenv PATH
if [ $(uname -m) = "x86_64" ]; then
FEATURES="cuda,asm"
else
FEATURES="cuda"
fi

for bench in {{benches}}; do
cargo criterion --bench $bench --features $FEATURES --message-format=json > "$bench-{{commit}}".json
done%
```
- conclusion : I always want the ASM feature, and I most of the time also want json output.
- one thing in the wishlist : if I run one of the jobs that have `message-format=json` I nearly certainly also want `XX_BENCH_OUTPUT = commit-comment`

[^1]: `cargo bench --bench foo -- --save-baseline foo_at_main`

Guida per i contributori

Nessuna guida per i contributori indicizzata per questo repository

Direzione di ricerca

Inizia individuando il justfile del repository e le definizioni esistenti dei job gpu-bench-ci e bench; confronta i relativi feature flag, la gestione di CUDA e i comandi di output. Il lavoro è completato quando i due job comparativi richiesti forniscono i valori predefiniti indicati per asm, CUDA, JSON, la denominazione dei file e BENCH_OUTPUT, preservando al contempo l'invocazione esistente del benchmark.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Valutazione

Stack tecnologico
github-actions, rust, shell
Ambito
ci-cd, performance
Tipo di issue
Funzionalità
Difficoltà
4/5
Tempo stimato
3-5 giorni
Stato di attività
Ferma
Chiarezza
Abbastanza chiara
Idoneità per principianti
30/100

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.