microsoft / microsoft/ebpf-for-windows

Automatically file bugs if BPF performance changes by more than 2σ

Open
#3,981 2 comments 1 reaction 2 assignees Assigned to @Alan-Jowett View on GitHub
P2 tests triaged
Dominant language
C
Stars
3.6k
Forks
311
Avg merge
6d 10h
Merged PRs (30d)
21

Description

This issue tracks adding **automatic monitoring + issue filing** for the perf metrics shown on the Perf Dashboard.

Perf results are already uploaded to Postgres and a regression query is already executed in CI (see `.github/workflows/upload-perf-results.yml`), but:

- There is no *scheduled* workflow that runs independently to catch regressions / detect “no recent data”.
- There is no automation to create/update GitHub issues per regressed metric.

Proposal doc: `docs/PerfRegressionAutoIssueFilingProposal.md`

---

## Requirements

- Create a **scheduled workflow** that executes the regression query.
- For each metric outside threshold, either file a new issue or update the currently active issue.
- If the most recent run is more than 3 days old, file a new issue or update an existing issue.

---

## Existing building blocks

- `.github/workflows/upload-perf-results.yml` already:
- Logs into Azure and fetches Postgres connection secrets from KeyVault.
- Downloads and runs `check_perf_results.sql` from `microsoft/bpf_performance`.
- Produces `results/regression_results.csv`.

- `check_perf_results.sql` parameters:
- `platform` (default `Windows 2019`)
- `repository` (default `microsoft/ebpf-for-windows`)
- `look_back` (default `30 days`)
- `max_sigma` (default `2`)

- `check_perf_results.sql` output columns:
- `timestamp`, `metric`, `value`, `mean_value`, `stddev_value`

Related noise sensitivity issue: #4115.

---

## Concrete plan (implementable)

### 1) Add scheduled workflow: `.github/workflows/monitor-perf-regressions.yml`

- Trigger:
- `schedule` (e.g. every 6 hours or daily)
- `workflow_dispatch`
- Must be independent from perf upload workflows.
- Use the same Azure KeyVault secrets used by `upload-perf-results.yml`:
- `PGDATABASE`, `PGHOST`, `PGUSER`, `PGPASSWORD`, `PGPORT`
- Run on `ubuntu-latest`.

### 2) Query regressions per platform

- Configure a platform list (same platforms used by perf runs), e.g.:
- `Windows 2019`
- `Windows 2022`

For each platform:

```sh
curl https://raw.githubusercontent.com/microsoft/bpf_performance/refs/heads/main/scripts/check_perf_results.sql > check_perf_results.sql
psql -f ./check_perf_results.sql \
-v platform='Windows 2019' \
-v repository='microsoft/ebpf-for-windows' \
-v look_back='30 days' \
-v max_sigma='3' \
--csv > regression_results.csv
```

### 3) Detect stale data (no recent results)

For each platform, run:

```sql
SELECT MAX("timestamp") AS last_run
FROM benchmarkresults
WHERE platform = :'platform'
AND repository = :'repository';
```

If `NOW() - last_run > INTERVAL '3 days'`, upsert a single issue per platform:

- Title: `Perf: no recent results for `
- Body: include the last seen timestamp and the dashboard link.

### 4) Upsert issues per (platform, metric)

To prevent duplicates, embed a stable marker in the issue body:

```text

```

Upsert behavior:

- If an open issue exists with the marker:
- Add a new comment containing latest timestamp/value/mean/stddev (+ computed z-score and percent delta).
- Else:
- Create a new issue:
- Title: `Perf regression: : `
- Labels: `tests` (and optionally a new `perf-regression` label)
- Body: dashboard link + stats table + marker.

Implementation suggestion:

- Use `actions/github-script` or a small Python script invoked by the workflow to:
- Parse `regression_results.csv`.
- Search open issues for the marker.
- Create issues/comments.

### 5) Reduce false positives / noise

The default `max_sigma=2` can be too sensitive (#4115).

MVP recommended policy (configurable via workflow env vars):

- Use `max_sigma=3` by default in the scheduled monitor workflow.
- Add optional minimum effect size gating before filing/updating:
- `abs_percent_delta = 100 * |value-mean| / |mean| >= MIN_PERCENT_DELTA` (e.g. 5%).

Optional follow-up: require the same regression to appear in 2 consecutive scheduled runs before filing.

---

## Deliverables checklist

- [ ] New scheduled workflow `.github/workflows/monitor-perf-regressions.yml`
- [ ] Per-platform stale-run detection and issue upsert (last run > 3 days)
- [ ] Regression query execution per platform and CSV parsing
- [ ] Per-(platform, metric) issue upsert using the embedded marker
- [ ] Noise controls (configurable `max_sigma`, optional percent-delta gating)
- [ ] Documentation: `docs/PerfRegressionAutoIssueFilingProposal.md`

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.