microsoft / microsoft/ebpf-for-windows
Automatically file bugs if BPF performance changes by more than 2σ
- Dominant language
- C
- Stars
- 3.6k
- Forks
- 311
- Avg merge
- 6d 10h
- Merged PRs (30d)
- 21
Description
This issue tracks adding **automatic monitoring + issue filing** for the perf metrics shown on the Perf Dashboard.
Perf results are already uploaded to Postgres and a regression query is already executed in CI (see `.github/workflows/upload-perf-results.yml`), but:
- There is no *scheduled* workflow that runs independently to catch regressions / detect “no recent data”.
- There is no automation to create/update GitHub issues per regressed metric.
Proposal doc: `docs/PerfRegressionAutoIssueFilingProposal.md`
---
## Requirements
- Create a **scheduled workflow** that executes the regression query.
- For each metric outside threshold, either file a new issue or update the currently active issue.
- If the most recent run is more than 3 days old, file a new issue or update an existing issue.
---
## Existing building blocks
- `.github/workflows/upload-perf-results.yml` already:
- Logs into Azure and fetches Postgres connection secrets from KeyVault.
- Downloads and runs `check_perf_results.sql` from `microsoft/bpf_performance`.
- Produces `results/regression_results.csv`.
- `check_perf_results.sql` parameters:
- `platform` (default `Windows 2019`)
- `repository` (default `microsoft/ebpf-for-windows`)
- `look_back` (default `30 days`)
- `max_sigma` (default `2`)
- `check_perf_results.sql` output columns:
- `timestamp`, `metric`, `value`, `mean_value`, `stddev_value`
Related noise sensitivity issue: #4115.
---
## Concrete plan (implementable)
### 1) Add scheduled workflow: `.github/workflows/monitor-perf-regressions.yml`
- Trigger:
- `schedule` (e.g. every 6 hours or daily)
- `workflow_dispatch`
- Must be independent from perf upload workflows.
- Use the same Azure KeyVault secrets used by `upload-perf-results.yml`:
- `PGDATABASE`, `PGHOST`, `PGUSER`, `PGPASSWORD`, `PGPORT`
- Run on `ubuntu-latest`.
### 2) Query regressions per platform
- Configure a platform list (same platforms used by perf runs), e.g.:
- `Windows 2019`
- `Windows 2022`
For each platform:
```sh
curl https://raw.githubusercontent.com/microsoft/bpf_performance/refs/heads/main/scripts/check_perf_results.sql > check_perf_results.sql
psql -f ./check_perf_results.sql \
-v platform='Windows 2019' \
-v repository='microsoft/ebpf-for-windows' \
-v look_back='30 days' \
-v max_sigma='3' \
--csv > regression_results.csv
```
### 3) Detect stale data (no recent results)
For each platform, run:
```sql
SELECT MAX("timestamp") AS last_run
FROM benchmarkresults
WHERE platform = :'platform'
AND repository = :'repository';
```
If `NOW() - last_run > INTERVAL '3 days'`, upsert a single issue per platform:
- Title: `Perf: no recent results for `
- Body: include the last seen timestamp and the dashboard link.
### 4) Upsert issues per (platform, metric)
To prevent duplicates, embed a stable marker in the issue body:
```text
```
Upsert behavior:
- If an open issue exists with the marker:
- Add a new comment containing latest timestamp/value/mean/stddev (+ computed z-score and percent delta).
- Else:
- Create a new issue:
- Title: `Perf regression: : `
- Labels: `tests` (and optionally a new `perf-regression` label)
- Body: dashboard link + stats table + marker.
Implementation suggestion:
- Use `actions/github-script` or a small Python script invoked by the workflow to:
- Parse `regression_results.csv`.
- Search open issues for the marker.
- Create issues/comments.
### 5) Reduce false positives / noise
The default `max_sigma=2` can be too sensitive (#4115).
MVP recommended policy (configurable via workflow env vars):
- Use `max_sigma=3` by default in the scheduled monitor workflow.
- Add optional minimum effect size gating before filing/updating:
- `abs_percent_delta = 100 * |value-mean| / |mean| >= MIN_PERCENT_DELTA` (e.g. 5%).
Optional follow-up: require the same regression to appear in 2 consecutive scheduled runs before filing.
---
## Deliverables checklist
- [ ] New scheduled workflow `.github/workflows/monitor-perf-regressions.yml`
- [ ] Per-platform stale-run detection and issue upsert (last run > 3 days)
- [ ] Regression query execution per platform and CSV parsing
- [ ] Per-(platform, metric) issue upsert using the embedded marker
- [ ] Noise controls (configurable `max_sigma`, optional percent-delta gating)
- [ ] Documentation: `docs/PerfRegressionAutoIssueFilingProposal.md`
Contributor guide
Assessment
This issue has not been assessed yet.