lincc-frameworks / lincc-frameworks/python-project-template

Fail asv workflows when performance deteriorates

Open
#252 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

enhancement new option
Dominant language
Jinja
Stars
85
Forks
16
Avg merge
6h 18m
Merged PRs (30d)
1

Description

Fail the ASV workflows if at least one of benchmarks deteriorates above a certain threshold (e.g. 150% / 200%).

**asv-main.yml**:
- Execute `asv run` without `|| true`, allowing the command to exit with a non-zero return code
- Submit benchmarks and deploy dashboard on gh-pages, if the previous command succeeded or not

**asv-pr.yml**:
- Execute `asv continuous` without `|| true`, allowing the command to exit with a non-zero return code
- Output the performance comment on the PR if the comparison was successful or not

**asv-nightly.yml**
- Execute `asv continuous` without `|| true`, allowing the command to exit with a non-zero return code
- Write the commit hash to cache, if the previous if the comparison was successful or not

More information about "on failure" executions [here](https://docs.github.com/en/actions/learn-github-actions/expressions#status-check-functions).

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with .github/workflows/asv-main.yml, asv-pr.yml, and asv-nightly.yml, and inspect how each currently handles asv run or asv continuous failures. Update the workflow failure conditions while preserving the requested submit, dashboard, PR comment, and cache steps, then verify the workflows still perform those steps when comparisons fail.

Written by the indexing model from the issue text.

Assessment

Tech stack
github-actions, python
Domain
ci-cd, devops, performance
Issue type
Feature
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Clearly specified
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.