firebase / firebase/firebase-tools

Cloud Functions v2: firebase deploy sends UpdateFunction twice in one deploy; duplicate fails with 409 ABORTED "unable to queue the operation"

Open
#10,731 3 comments 3 reactions 0 assignees View on GitHub
api: functions reproducible type: bug
Dominant language
TypeScript
Stars
4.5k
Forks
1.3k
Avg merge
1d 12h
Merged PRs (30d)
84

Description

### [REQUIRED] Environment info

- **firebase-tools:** 15.22.3
- **Platform:** Linux (GitHub Actions, ubuntu-24.04), Node.js 22
- **Auth:** service account via `GOOGLE_APPLICATION_CREDENTIALS` / ADC (non-interactive CI)

### [REQUIRED] Test case / Steps to reproduce

1. Have a deployed 2nd-gen (Cloud Functions v2 / Cloud Run) HTTPS function.
2. Make a real change so the function is not skipped.
3. Run `firebase deploy --only functions` (non-interactive, service-account auth).

### [REQUIRED] Expected behavior

A single `UpdateFunction` operation per changed function, and the deploy reports success.

### [REQUIRED] Actual behavior

The CLI issues `google.cloud.functions.v2.FunctionService.UpdateFunction` **twice for the same function, ~2 seconds apart, within a single `firebase deploy`**. The first call is accepted and runs the deploy operation to completion; the second (duplicate) call is rejected by the API with `code 10 ABORTED` / HTTP 409 `unable to queue the operation`, because the first operation is still in flight.

The CLI surfaces the *second* call's 409 and reports the deploy as failed — **even though the function actually deployed successfully** (it reaches `ACTIVE` with the new source hash, and a new Cloud Run revision goes live).

A subsequent re-run reports `Skipping the deploy of unchanged functions` / `No changes detected` (because the first call already deployed), which masks the problem and makes it look like a transient/flaky 409.

#### CLI output

```
i functions: updating Node.js 22 (2nd Gen) function MY_FUNCTION(us-central1)...
⚠ functions: Request to https://cloudfunctions.googleapis.com/v2/projects/MY_PROJECT/locations/us-central1/functions/MY_FUNCTION?updateMask=name%2C...%2Clabels had HTTP Error: 409, unable to queue the operation
⚠ functions: failed to update function projects/MY_PROJECT/locations/us-central1/functions/MY_FUNCTION
```

#### Evidence (Cloud Audit Logs)

All entries are from the same principal, same caller IP, and the same `FirebaseCLI/15.22.3` user agent — i.e. **one CLI invocation**. Project/SA identifiers redacted.

| Time (UTC) | `methodName` | Result |
|---|---|---|
| `02:43:37.582` | `…v2.FunctionService.UpdateFunction` (**request A**) | accepted → starts `operation-…-6eff80c7` (`operation.first = true`, empty `status`) |
| `02:43:39.523` | `…v2.FunctionService.UpdateFunction` (**request B**) | `status.code: 10`, `"unable to queue the operation"` (severity ERROR) |
| `02:44:25.434` | operation `…-6eff80c7` completes | function `state: ACTIVE`, new hash live, `operation.last = true` |

Supporting `run.googleapis.com` system events show the underlying Cloud Run revision rolling out and reporting **"Deploying revision succeeded in 6.56s"** at `02:44:23`, confirming request A's operation deployed the function correctly.

Key point: requests **A** and **B** are identical `UpdateFunction` calls (same `update_mask`, same `function` body, same `firebase-functions-hash`) ~1.9s apart from the same CLI process. **B is a redundant duplicate.**

### Impact

- Every real-change deploy of a 2nd-gen function spuriously fails in CI, despite the deploy actually succeeding.
- The failure looks transient (a re-run says "No changes detected"), so it is easy to misdiagnose as a Google-side or IAM issue. (`cloudfunctions.functions.update` was `granted: true` on both calls — it is not a permissions problem.)

### Likely cause

Appears to be a client-side retry re-sending `UpdateFunction` after the first request was already accepted server-side, with no idempotency guard — so the retry collides with its own in-flight operation and gets `ABORTED`.

### Workaround

Wrap the deploy and retry on `unable to queue the operation`; the retry observes "No changes detected" (because the first call already deployed) and the deploy reports success.

### Related

- #10661 (`firebase deploy exits 0 after "409 unable to queue the operation"`)
- #6989 (`firebase deploy returns exit code 0 even if a function failed to be deployed`)

These cover the exit-code / 409-handling symptoms; this report focuses on the **duplicate `UpdateFunction` submission** that produces the 409 in the first place.

Contributor guide

Open the contributing guide

Research direction

Start by reproducing `firebase deploy --only functions` for a changed Cloud Functions v2 function and trace the `FunctionService.UpdateFunction` request and retry path. Compare the two identical requests and their timing in the audit evidence; done means one update operation is submitted and the deploy reports success without a duplicate 409.

Written by the indexing model from the issue text.

Assessment

Tech stack
gcp, typescript
Domain
cli, cloud
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.