cockroachdb / cockroachdb/cockroach

roachtest: preserve failed GitHub issue posts during GitHub outages

Open
#169,884 1 comment 0 reactions 1 assignee Claimed by @williamchoe3 View on GitHub
A-testeng-infra C-enhancement T-testeng
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

## Summary

Roachtest currently treats GitHub issue creation as the durable record for test failures. When GitHub is unavailable or issue posting fails due to some transient issue, roachtest does not emit that failure and it isn't directly reported unless someone notices the test failure through Teamcity or some other reporting means.

This makes GitHub a reliability bottleneck for reporting roachtest failures, which is important as GitHub's reliability has decreased. Also when considering rare failure modes, we shouldn't be taking any loss of test reporting for granted.

Currently, when `roachtest` fails to post to GitHub, we simply exit with a nonzero exit code and test-eng is alerted. The following aims to take the next step and serialize these requests so they can be retried or replayed at a later time.

## Proposed direction

The current implementation direction is a GCS-backed dead letter queue for failed roachtest GitHub issue posts:
- Roachtest writes failed GitHub issue post requests into a GCS DLQ bucket.
- Entries are stored under `failed/`, claimed under `processing/`, and archived under `processed/`.
- A replay binary reads DLQ entries, reconstructs the GitHub issue post request, and calls the existing `issues.Post` path.
- GCS object preconditions protect against two operators claiming the same failed entry concurrently.
- Infra provides a manually triggered Cloud Run Job for replay after a GitHub outage.
- The Cloud Run Job reads a GitHub API token from Secret Manager as `GITHUB_API_TOKEN`.

## Current implementation work

App-side PR:
- https://github.com/cockroachdb/cockroach/pull/167064
- Adds the GCS DLQ writer and `dlq-replay` CLI.
- Wraps roachtest's existing issue posting path when `GITHUB_DLQ_BUCKET` is set.
- Serializes failed issue post requests and failure context to GCS.
- Replays entries by reconstructing `issues.PostRequest` / `issues.Options`.

Infra PR:
- https://github.com/cockroachlabs/crl-infrastructure/pull/5448
- Adds the GCP infra in `cockroach-testeng-infra`.
- Creates the DLQ bucket, Artifact Registry repo, Cloud Run Job, replay service account, and IAM.
- Uses Secret Manager for `github-dlq-replay-token`, created and rotated out-of-band.

Supplemental design notes:
- Infra decision doc: https://docs.google.com/document/d/1Sovj2mUu3eAUDqHuXEgxpZRZxPIZ1Acuhuj1pFadYAE/edit?tab=t.0

## Post implementation work

Enable in CI through env var
Get GitHub token from dev-inf: [slack thread](https://cockroachlabs.slack.com/archives/CJ0H8Q97C/p1779809513405309)

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.