Comfy-Org / Comfy-Org/ComfyUI_frontend

Alert (Slack) on backport/release failures + assert all needs-backport PRs landed

Open
#14,038 1 comment 1 reaction 0 assignees View on GitHub
area:CI/CD enhancement
Dominant language
TypeScript
Stars
2k
Forks
699
Avg merge
1d 7h
Merged PRs (30d)
490

Description

## Problem

During the 1.47 cycle **two auto-backports failed silently** (#13875, #13971) and core **v1.47.9 shipped without those fixes**. There was no alert — the failures were only discoverable by noticing a comment on an already-merged PR.

## What already exists (audited on `origin/main`)

- `pr-backport.yaml` **does** post a **PR comment** cc-ing the author on a failed backport (conflicts / branch-missing / push-failed / branch-create-failed), gated on `failure() && steps.backport.outputs.failed`.
- **Why that's insufficient:** the source PR is already **merged**, so a comment on it is easily missed. There is **no** push notification to a release/eng channel, and **no** assertion at release time that every `needs-backport` PR actually landed on the release branch. That's exactly how #13875 / #13971 slipped.
- `backport-auto-merge.yaml` (`*/15` cron) merges eligible backport PRs and reconciles races, but has **no failure alerting**.
- `coverage-slack-notify.yaml` already establishes a **reusable Slack pattern** in this repo: `SLACK_BOT_TOKEN` secret → `chat.postMessage` to a channel. This is the plumbing to reuse.
- **Overlap with #14033:** that issue covers the release-automation *reliability* fixes (the `publish-pypi` tag-wait timeout, the cosmetic `release-draft-create` failure). This issue is the **observability half** — alerting so failures are *seen* — and should be done alongside it, not duplicate its reliability changes.

## Proposal

1. **Slack alert on backport failure.** In `pr-backport.yaml`, on `failure()` (or when `steps.backport.outputs.failed` is non-empty), post to the release/eng Slack channel in addition to the PR comment, reusing the `coverage-slack-notify.yaml` `SLACK_BOT_TOKEN` + `chat.postMessage` pattern. Include the source PR, target branch(es), and failure reason.
2. **Release-done backport assertion.** Add a check (in the release workflow, or a scheduled reconciliation) that asserts every merged PR labeled `needs-backport` has a corresponding landed backport on the target release branch before/at release cut — and Slack-alerts on any gap. This closes the "shipped v1.47.9 without the fixes" hole.
3. Optionally extend the same Slack alert to `backport-auto-merge.yaml` failures and to release-workflow failures generally (dovetails with #14033).

## Acceptance criteria

- [ ] A failed auto-backport posts a Slack alert to the release/eng channel (not only a PR comment).
- [ ] A release cannot silently ship with an outstanding `needs-backport` PR that never landed — a reconciliation check flags/alerts the gap.
- [ ] Alerts include enough context (source PR, target branch, reason) to act without digging.
- [ ] Reuses the existing `SLACK_BOT_TOKEN` / `chat.postMessage` pattern rather than adding new secrets/infra.

## Motivating incident

Silent backport failures #13875 and #13971 → core v1.47.9 shipped without the fixes. Companion to release-reliability issue #14033.

Contributor guide

Open the contributing guide

Research direction

Start by reading .github/workflows/pr-backport.yaml and coverage-slack-notify.yaml to trace the existing failure path and reusable SLACK_BOT_TOKEN/chat.postMessage pattern. Then locate the release workflow and inspect backport-auto-merge.yaml. Done means failed backports and missing needs-backport landings produce actionable Slack alerts with the source PR, target branch, and reason.

Written by the indexing model from the issue text.

Assessment

Tech stack
github-actions
Domain
ci-cd, observability, release
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.