con / con/git-annex

somehow organize performance benchmarking of git-annex

Open
#3 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
29
Forks
5
PR merge metrics
No merged PRs in 30d

Description

We added some checks to git-annex build workflow to spot some cases which could lead to slow(er) standalone build operation, but overall we do not have a good way to detect whenever git-annex "slow downs". We can only see reflection of that whenever we try a new snapshot build sweeping through our datalad tests but then it becomes an archeological expedition to see which change brought the pessimization.

It would be nice to establish automated and consistent benchmarking of git-annex builds as pertinent to datalad.

Proposal:

- take some release of datalad still compatible with current annex build (so we could take current release ATM IIRC)
- use asv benchmarks of that datalad but for benchmarking git-annex (so whenever we improve our datalad benchmarks collection, it automagically helps to benchmark git-annex)
- establish `datalad/git-annex-benchmarking` on github
- `git subtree` benchmarks from datalad
- Include git-annex's master branch (from git://git.kitenet.net/git-annex) as `annex-master` branch known to that repo
- I think asv can benchmark commits in another branch, while benchmarks would be in the `master`. So asv configuration would do that
- add pythonish setup to `make standalone install` the git-annex so asv could deploy any given version of git-annex ( I wonder if there is smth like ccache for haskell ;))
- I think we better right away establish a singularity container based on e.g. https://github.com/datalad-tester/dev-containers/blob/master/Singularity.10.20200209.1 which would add `apt build-dep git-annex-standalone`, and that would be container to run asv in -- it would have all build-dependencies etc, and we use this for "worker" env (see below) later
- add github action to run on cron, which would
- `git checkout annex-master && git pull --ff-only && git push origin annex-master && git checkout master`
- `asv run` on new commits in `annex-master` and then `asv gh-pages && datalad save -m "ASV results update" .asv && git push origin`
- provide github actions worker on a dedicated box (I have some, consistent timing), probably within singularity (at least some isolation and again -- consistency)
- make that github action to run only only on pushes, not PRs so we do not anyhow compromise security

WDYT @mih @kyleam @bpoldrack @jwodder

FYI @joeyh

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.