Comfy-Org / Comfy-Org/ComfyUI_frontend

CI: intermittent TLS interception (DEPTH_ZERO_SELF_SIGNED_CERT) on Azure westus runners — 23/23 failures in one region

Open Beginner friendly
#15,134 2 comments 1 reaction 0 assignees View on GitHub
area:CI/CD
Dominant language
TypeScript
Stars
2k
Forks
699
Avg merge
1d 3h
Merged PRs (30d)
495

Description

## Summary

Since `2026-08-10T22:44Z`, jobs on GitHub-hosted runners have been intermittently failing TLS handshakes to `github.com` / `api.github.com` with a self-signed-certificate error. **Every single occurrence is on a runner provisioned in Azure region `westus`.**

This is a platform-side problem — nothing in this repo can cause it and no repo change can fix it — but one file here turns it into a hard job failure, and that part we can fix.

## Symptom

Two different TLS stacks, same failure.

Node 24 / undici, inside `actions/github-script@v9` (19 of 23 occurrences):

```
GET /repos/Comfy-Org/ComfyUI_frontend/issues/15114/comments - 500 with id UNKNOWN in 53ms
##[error]Unhandled error: HttpError: self-signed certificate; if the root CA is installed locally, try running Node.js with --use-system-ca
url: 'https://api.github.com/repos/Comfy-Org/ComfyUI_frontend/issues/15114/comments',
[cause]: TypeError: fetch failed
[cause]: Error: self-signed certificate
at TLSSocket.onConnectSecure (node:internal/tls/wrap:1748:34)
code: 'DEPTH_ZERO_SELF_SIGNED_CERT'
```

`git` / libcurl / OpenSSL, during `actions/checkout` (4 of 23) — job `94006671059`, `validate-fonts`, 04:06:59Z, westus. Retried 3× over 30s, failed every time:

```
##[error]fatal: unable to access 'https://github.com/Comfy-Org/ComfyUI_frontend/':
server certificate verification failed. CAfile: none CRLfile: none
```

The chain terminates at depth 0 (self-signed leaf), i.e. the connection is being intercepted. The HTTP `500` is octokit mislabeling a transport failure.

## Evidence: it is `westus`, with a control

Swept every failed job in the repo for Aug 9–12 UTC (432 + 257 + 191 failed jobs) and pulled check-run annotations.

| Date (UTC) | TLS-symptom jobs | Region |
| --- | --- | --- |
| Aug 9–10 | 1 (first at `2026-08-10T22:44:55Z`) | westus |
| Aug 11 | 11 | westus ×11 |
| Aug 12 (to 06:43Z) | 11 | westus ×11 |

**23 of 23 in `westus`.**

Base-rate control: sampling region on *successful* jobs in the same windows gives westus = 14/80 (17.5%) at 04:00–05:00Z and 9/60 (15%) at 05:42–06:43Z. So westus is ~1-in-6 of normal traffic but 23-of-23 of the failures. Random assignment is ruled out by many orders of magnitude.

It is **intermittent**, not continuous, within the region — an in-run control: job `94010344278` (`changes`, run 31563449217) ran in westus at 04:30:04Z and cloned over HTTPS fine, 109 seconds before `deploy-and-comment` in the *same run* failed.

Ruled out as the common factor:

- **Runner image** — spans `20260720.247.2` (21) and `20260810.271.1` (2).
- **Provisioner** — spans `20260707.563` and `20260729.566`.
- **A single bad VM** — every failing job has a distinct `runner_name`.
- **Action version** — 19 Node/undici vs 4 git/OpenSSL. Two unrelated TLS implementations fail identically.
- **Endpoint** — both `github.com` and `api.github.com`.
- **Repo config** — no `HTTP(S)_PROXY`, no `NODE_EXTRA_CA_CERTS`, no self-hosted runners, no containers on the affected jobs.

### Not the same as the artifact ECONNRESETs

Separately, `94015375556` (shard 1/16, 05:03Z, **eastus2**, in a container) and `93969325990` (shard 2/16, 00:16Z, **westus3**) failed with:

```
##[error]Unable to download artifact(s): Failed to ListArtifacts: Unable to make request: ECONNRESET
```

Different region, different endpoint (artifact/blob service), different error class, no certificate involved. Same broad symptom class, **not** demonstrated to be the same root cause. Filing them together would be unsupported.

## Recent occurrences (Aug 12 UTC)

`93968741869` deploy-and-comment/Storybook 00:13 · `93969420543` deploy-and-comment-forked-pr 00:17 · `93976619942` comment/Unified Report 00:58 · `93976804925` deploy-and-comment-forked-pr 00:59 · `93982015885` changes/Perf Report 01:30 · `93989973195` changes/Tests Unit 02:19 · `94006671059` validate-fonts 04:06 · `94010567519` (PR #15115) 04:31 · `94014303389` (PR #15114) 04:54 · `94022091028` shard 8/16 05:42 · `94029192656` changes/Website E2E 06:23

## Impact

`deploy-and-comment` is **not** a required check — required contexts on `main` are `cla-assistant` (ruleset 18070802) and `test`, `lint-and-format`, `e2e-status`, `website-e2e` (ruleset 991238). So today this is red-check noise rather than a merge blocker. But it has also hit `validate-fonts`, `Tests Unit`, `Website E2E` and a Playwright shard, which are closer to the required set.

## Proposed local mitigation

`.github/actions/upsert-comment-section/action.yaml:29` calls `actions/github-script@v9` with **no retry and no `continue-on-error`**. A purely cosmetic PR-comment update therefore hard-fails an entire job on one intercepted handshake. That single file is the highest-leverage fix available to us:

1. Add `continue-on-error: true` to the comment-upsert step — a failed comment update should never fail a build.
2. Optionally wrap the octokit call in a small retry with backoff.

Neither addresses the root cause, which is on GitHub's side.

## To confirm root cause

A one-line diagnostic step (`openssl s_client -connect api.github.com:443 &1 | head -30` or `curl -vI https://api.github.com`) run on failure would capture the served leaf certificate and identify the interception issuer. Worth adding temporarily given this is escalating (1/day → ~11/day).

Contributor guide

Open the contributing guide

Research direction

Start with .github/actions/upsert-comment-section/action.yaml at line 29 and inspect the comment-upsert step using actions/github-script@v9. Confirm how a failed GitHub API request currently affects the job, then verify that a transient TLS failure in this cosmetic update no longer fails the build; any retry should remain small and scoped to this step.

Written by the indexing model from the issue text.

Assessment

Tech stack
github-actions
Domain
ci-cd
Issue type
Bug
Difficulty
1/5
Estimated time
Under an hour
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
78/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.