cockroachdb / cockroachdb/cockroach
roachtest: preserve failed GitHub issue posts during GitHub outages
- Dominant language
- Go
- Stars
- 32.5k
- Forks
- 4.1k
- PR merge metrics
- PR metrics pending
Description
## Summary
Roachtest currently treats GitHub issue creation as the durable record for test failures. When GitHub is unavailable or issue posting fails due to some transient issue, roachtest does not emit that failure and it isn't directly reported unless someone notices the test failure through Teamcity or some other reporting means.
This makes GitHub a reliability bottleneck for reporting roachtest failures, which is important as GitHub's reliability has decreased. Also when considering rare failure modes, we shouldn't be taking any loss of test reporting for granted.
Currently, when `roachtest` fails to post to GitHub, we simply exit with a nonzero exit code and test-eng is alerted. The following aims to take the next step and serialize these requests so they can be retried or replayed at a later time.
## Proposed direction
The current implementation direction is a GCS-backed dead letter queue for failed roachtest GitHub issue posts:
- Roachtest writes failed GitHub issue post requests into a GCS DLQ bucket.
- Entries are stored under `failed/`, claimed under `processing/`, and archived under `processed/`.
- A replay binary reads DLQ entries, reconstructs the GitHub issue post request, and calls the existing `issues.Post` path.
- GCS object preconditions protect against two operators claiming the same failed entry concurrently.
- Infra provides a manually triggered Cloud Run Job for replay after a GitHub outage.
- The Cloud Run Job reads a GitHub API token from Secret Manager as `GITHUB_API_TOKEN`.
## Current implementation work
App-side PR:
- https://github.com/cockroachdb/cockroach/pull/167064
- Adds the GCS DLQ writer and `dlq-replay` CLI.
- Wraps roachtest's existing issue posting path when `GITHUB_DLQ_BUCKET` is set.
- Serializes failed issue post requests and failure context to GCS.
- Replays entries by reconstructing `issues.PostRequest` / `issues.Options`.
Infra PR:
- https://github.com/cockroachlabs/crl-infrastructure/pull/5448
- Adds the GCP infra in `cockroach-testeng-infra`.
- Creates the DLQ bucket, Artifact Registry repo, Cloud Run Job, replay service account, and IAM.
- Uses Secret Manager for `github-dlq-replay-token`, created and rotated out-of-band.
Supplemental design notes:
- Infra decision doc: https://docs.google.com/document/d/1Sovj2mUu3eAUDqHuXEgxpZRZxPIZ1Acuhuj1pFadYAE/edit?tab=t.0
## Post implementation work
Enable in CI through env var
Get GitHub token from dev-inf: [slack thread](https://cockroachlabs.slack.com/archives/CJ0H8Q97C/p1779809513405309)
Contributor guide
Assessment
This issue has not been assessed yet.