firebase / firebase/firebase-tools
Functions v2: after 409 "unable to queue the operation", retry deadlocks on SourceTokenScraper.getToken() and firebase deploy exits 0 mid-deploy (silent CI false-green)
- Dominant language
- TypeScript
- Stars
- 4.5k
- Forks
- 1.3k
- Avg merge
- 1d 12h
- Merged PRs (30d)
- 84
Description
### [REQUIRED] Environment info
**firebase-tools:** 15.22.4 (installed fresh via `npm i -g firebase-tools`; the code path below is verified against the v15.22.4 source and is also present in v15.19.0)
**Platform:** Linux (GitHub Actions Ubuntu runner), Node.js 22.23.0, non-interactive CI with service-account ADC
### [REQUIRED] Test case
A project with an already-deployed 2nd-gen function (here: a Python 3.12 gen2 function) where `firebase deploy --only functions` receives HTTP 409 `unable to queue the operation` on the **first** `UpdateFunction` attempt. The `functionsv2deployoptimizations` experiment must be enabled — it is [on by default](https://github.com/firebase/firebase-tools/blob/v15.22.4/src/experiments.ts#L34-L41).
The 409 itself can come from the CLI's own `apiv2` client re-sending the non-idempotent PATCH after a connection-level "premature close" (see "Where the 409 comes from" below — same duplicate-submission failure mode as #10731), or from a concurrent deploy to the same project. This issue is about what the CLI does *after* any such 409: instead of retrying, it deadlocks and exits `0` mid-deploy with no error.
### [REQUIRED] Steps to reproduce
Natural repro (flaky by nature — depends on the server returning the 409):
1. Change one 2nd-gen function and run `firebase deploy --only functions --non-interactive` in CI (GitHub-hosted runner; Node 22.23.0 makes the underlying premature-close error more frequent).
2. On the unlucky runs, the `UpdateFunction` PATCH's response connection dies ("premature close" on a stale keep-alive socket); `apiv2` re-sends the PATCH ~1–2 s later, and the duplicate gets `409, unable to queue the operation` because the first attempt's LRO is already in flight (evidence below).
3. The process exits with **code 0** exactly ~40 s after the 409 warning, mid-deploy, with no further output.
Deterministic repro (fault injection):
1. Put an HTTPS-intercepting proxy in front of the CLI that answers the first `PATCH .../v2/projects/*/locations/*/functions/*` with `409 {"error":{"code":409,"status":"ABORTED","message":"unable to queue the operation"}}` and passes everything else through.
2. Run `firebase deploy --only functions` with one changed v2 function.
3. Process exits 0 after ~40 s: no `Deploy complete!`, no error summary.
Equivalent unit-level repro: call `fabricator.updateV2Function` with `gcfV2.updateFunction` mocked to reject once with a 409 `FirebaseError` — the returned promise never settles.
### [REQUIRED] Expected behavior
The 409 is retried (as designed — `DEFAULT_RETRY_CODES = [429, 409, 503]`), and the deploy either eventually succeeds or fails loudly with a non-zero exit code and an error summary.
### [REQUIRED] Actual behavior
The first retry deadlocks awaiting `SourceTokenScraper.getToken()`. The await can never resolve, Node's event loop is left with no timers or handles, and the process **exits naturally with code 0 mid-deploy** — no `Deploy complete!`, no error summary, no non-zero exit. In CI this makes a possibly-failed deploy look green.
Observed fingerprint — identical in two GitHub Actions runs ~15 minutes apart (2026-07-01, identifiers redacted):
```
i functions: updating Python 3.12 (2nd Gen) function FUNCTION_NAME(us-central1)...
⚠ functions: Request to https://cloudfunctions.googleapis.com/v2/projects/PROJECT_ID/locations/us-central1/functions/FUNCTION_NAME?updateMask=name%2CbuildConfig.runtime%2C...%2CbuildConfig.sourceToken%2C...%2Clabels had HTTP Error: 409, unable to queue the operation
⚠ functions: failed to update function projects/PROJECT_ID/locations/us-central1/functions/FUNCTION_NAME
```
…and then nothing. Runner timestamps:
| Run | 409 warning | Process exit (code 0) | Δ |
|---|---|---|---|
| A | 21:53:19.13 | 21:53:59 | ≈ 39.9 s |
| B | 22:08:19.53 | 22:08:59 | ≈ 39.5–40 s |
The ~40 s is not a coincidence: it is exactly the first retry backoff, `20000 * 2^1` (code walk below). In both runs the changed function was the only update in its codebase/changeset (everything else was `Skipped (No changes detected)`).
### Root cause (code walk against v15.22.4)
1. **409 is a retry code.** [`DEFAULT_RETRY_CODES = [429, 409, 503]`](https://github.com/firebase/firebase-tools/blob/v15.22.4/src/deploy/functions/release/executor.ts#L22); the queue [handler rethrows retryable errors](https://github.com/firebase/firebase-tools/blob/v15.22.4/src/deploy/functions/release/executor.ts#L34-L52) back to the throttler for retry.
2. **First retry fires at exactly 40 s.** The functions executor is built with [`retries: 30, backoff: 20000, maxBackoff: 100000`](https://github.com/firebase/firebase-tools/blob/v15.22.4/src/deploy/functions/release/index.ts#L82-L85); the throttler waits [`min(delay * 2^retryNumber, maxDelay)`](https://github.com/firebase/firebase-tools/blob/v15.22.4/src/throttler/throttler.ts#L19-L21) with `retryNumber = retryCount + 1 = 1` before [re-running the task](https://github.com/firebase/firebase-tools/blob/v15.22.4/src/throttler/throttler.ts#L272-L285) → `20000 * 2 = 40000 ms`. Matches the observed Δ in both runs.
3. **The retried closure re-runs `getToken()`.** The task the throttler re-executes is the closure in [`updateV2Function`](https://github.com/firebase/firebase-tools/blob/v15.22.4/src/deploy/functions/release/fabricator.ts#L568-L588), which begins with `apiFunction.buildConfig.sourceToken = await scraper.getToken()` (gated on the default-on `functionsv2deployoptimizations` experiment).
4. **`getToken()` can never resolve.** On the first attempt the scraper transitioned `NONE → FETCHING` and returned `undefined` ([sourceTokenScraper.ts#L33-L36](https://github.com/firebase/firebase-tools/blob/v15.22.4/src/deploy/functions/release/sourceTokenScraper.ts#L33-L36)). The PATCH then 409'd, so `pollOperation` — and with it [`scraper.poller`](https://github.com/firebase/firebase-tools/blob/v15.22.4/src/deploy/functions/release/sourceTokenScraper.ts#L69-L83), the only code that resolves the token promise with a value — never ran. On the retry, `getToken()` takes the `FETCHING` branch and [awaits `this.promise`](https://github.com/firebase/firebase-tools/blob/v15.22.4/src/deploy/functions/release/sourceTokenScraper.ts#L37-L43), a promise only `poller` or [`abort()`](https://github.com/firebase/firebase-tools/blob/v15.22.4/src/deploy/functions/release/sourceTokenScraper.ts#L29-L31) can resolve.
5. **`abort()` is unreachable.** It is only called in `updateV2Function`'s final [`.catch`](https://github.com/firebase/firebase-tools/blob/v15.22.4/src/deploy/functions/release/fabricator.ts#L584-L588), which requires the executor promise to reject — i.e. `RetriesExhaustedError` after all 30 retries. That never happens, because retry #1 is blocked awaiting the very promise that only this code path could release. Deadlock.
6. **Deadlock → silent exit 0.** The forever-pending `await` leaves nothing on the event loop (the 40 s backoff timer was the last live handle), so Node runs out of work and exits with code 0 mid-deploy.
`createV2Function` has the identical structure ([fabricator.ts#L415-L440](https://github.com/firebase/firebase-tools/blob/v15.22.4/src/deploy/functions/release/fabricator.ts#L415-L440)), so a 409 on the first *create* in a changeset should deadlock the same way. Any other functions in the same changeset that are awaiting the shared scraper promise hang with it — the process dies before their own retries could ever run.
### Where the 409 comes from: apiv2 re-sends non-idempotent writes after a "premature close" (same class as #10731)
The duplicate request that produces the 409 is the CLI's own `apiv2` client, not the server:
- [`isPrematureCloseError`](https://github.com/firebase/firebase-tools/blob/v15.22.4/src/apiv2.ts#L175-L193) matches `ERR_STREAM_PREMATURE_CLOSE`, `ECONNRESET`, `/premature close/i`, `/socket hang up/i`.
- On such an error, if the body is replayable, [apiv2 logs `*** [apiv2] retrying without keep-alive after a premature close error` and calls `operation.retry`](https://github.com/firebase/firebase-tools/blob/v15.22.4/src/apiv2.ts#L540-L562) ([`minTimeout: 1000, maxTimeout: 5000`](https://github.com/firebase/firebase-tools/blob/v15.22.4/src/apiv2.ts#L462-L467)) — **with no check that the HTTP method is idempotent**. A PATCH/POST whose request landed server-side but whose response connection died gets re-sent.
Sequence: PATCH attempt #1 lands and creates the update LRO → the response socket dies (stale keep-alive socket on a GitHub-hosted runner; the premature-close error is more frequent on Node 22.23.0, per the maintainer diagnosis in #10731) → apiv2 re-sends the PATCH ~1–2 s later → the duplicate gets `409 ABORTED, "unable to queue the operation"` because attempt #1's LRO is in flight.
Cloud Audit Logs match exactly: for the single logical client call, **two** `google.cloud.functions.v2.FunctionService.UpdateFunction` entries ~1.8 s apart — same principal, caller IP, and `FirebaseCLI` user agent. The first was accepted and created the update LRO; the second failed with `status.code 10` (ABORTED), "unable to queue the operation". The LRO completed successfully ~75 s later (function `ACTIVE` with the new source hash).
Corroboration from the same day, same project: a Firebase Hosting deploy (via FirebaseExtended/action-hosting-deploy, also firebase-tools 15.22.4, run with `--debug`) hit the identical pattern on the releases endpoint — the log shows
```
*** [apiv2] retrying https://firebasehosting.googleapis.com/v1beta1/projects/-/sites/SITE_ID/channels/live/releases?versionName=... without keep-alive after a premature close error
```
and the retried POST got `400 FAILED_PRECONDITION "the supplied version ... is the current active version"` — proving attempt #1 had landed. Same root flake, opposite symptom: **hosting fails loudly on its duplicate (false red), functions exits 0 silently via the deadlock (false green).**
So in these runs the 409 the CLI surfaced was spurious and the function happened to deploy anyway. But the CLI can't know that — and when the 409 is real (e.g. a concurrent deploy's operation in flight), nothing deploys and CI is still green, because the process dies before the retry that was supposed to handle exactly this case.
### Impact
- CI reports success for a deploy that never printed a result and may not have deployed anything — silently stale functions behind a green build.
- Plausibly explains part of #10661 (exit 0 after "409 unable to queue"; Hosting release never happens): if the functions phase deadlocks and the event loop drains, the process exits 0 before the Hosting release step ever runs. Note this is *not* the #6989/#10253 analytics-crash mechanism — this is a genuine deadlock in the functions retry path, still present in 15.22.4.
### Suggested fixes
1. **Release the scraper before rethrowing a retryable error.** In the task closures of `updateV2Function`/`createV2Function`, if this invocation moved the scraper `NONE → FETCHING` and the request/poll then failed, call `scraper.abort()` (or a new `scraper.release()`) before rethrowing. `abort()` resolves the promise with `{aborted: true}`; the `FETCHING` branch of `getToken()` then re-arms the promise and returns `undefined`, so the retry — and any other waiters — proceed token-less, which is the designed degraded path. (Resolving an already-settled promise is a no-op, so this is safe when the poller did run.)
2. **And/or a timeout inside `getToken()`** — race the promise against a deadline; on expiry, log and return `undefined`. Turns any future "lost token producer" variant into a slower deploy instead of a deadlock.
3. **Make apiv2's premature-close retry conflict-aware for non-idempotent writes.** Either don't blindly re-send PATCH/POST after a premature close, or treat the characteristic conflict on the re-send — `409 "unable to queue the operation"` on a functions update, `400 "…is the current active version"` on a hosting release — as probable success of the first attempt and verify/poll instead of failing. (Arguably its own bug; happy to split it out if preferred.)
4. **Hardening for the whole class:** during a deploy, a `process.on('beforeExit')` guard that exits non-zero if the event loop drains before the deploy summary is printed. #6989, #10253, #10661 and this issue are all "CLI dies silently with exit 0" variants — a drain-guard makes every future one loud.
### Related
- #10731 — the same duplicate-`UpdateFunction` failure mode; the maintainer diagnosis there points at Node 22.23.0's premature-close error, which is what trips apiv2's retry. This issue adds the client-side retry mechanics (apiv2.ts) and covers the CLI's handling of the 409 afterwards.
- #10661 — same observable outcome (exit 0 after a 409); the deadlock described here plausibly explains its "Hosting never released, no error, exit 0" behavior.
- #6989 / #10253 — earlier exit-0-on-failure bug; fixed, but via a different mechanism (analytics crash). This path is untouched by that fix.
Contributor guide
Research direction
Start with src/deploy/functions/release/fabricator.ts and sourceTokenScraper.ts, then trace the retry flow through executor.ts and throttler.ts. Run the unit-level repro with updateV2Function and updateFunction mocked to reject once with a 409. Done means the retry settles, the deploy either completes or reports an error with a non-zero exit, and the create path is covered too.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- node.js, typescript
- Domain
- cli, tooling
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Clearly specified
- Newbie friendliness
- 52/100