firebase / firebase/firebase-tools

functions: deploy exits 0 with endpoints left un-updated after 429 quota errors, and prints no error summary

Open
#11,040 2 comments 0 reactions 0 assignees View on GitHub
api: functions type: bug
Dominant language
TypeScript
Stars
4.5k
Forks
1.3k
Avg merge
1d 12h
Merged PRs (30d)
84

Description

### [REQUIRED] Environment info

**firebase-tools:** 15.22.2

**Platform:** Linux (ubuntu-latest GitHub Actions runner), Node 24.16.0

### [REQUIRED] Test case

A 2nd-gen functions codebase with ~81 endpoints, all in one region (`europe-west3`), deployed with:

```
firebase deploy --only functions --project --force
```

The codebase is a single bundle, so every endpoint's `sourceHash` changes on any source change and all ~81 are updated on every deploy. That exceeds `cloudfunctions.googleapis.com/v2_write_requests_regional`, which is 60 mutations/min/region and is documented as not increasable, so the deploy reliably takes HTTP 429s and leans on the executor's retries.

I have not reduced this to a minimal repro — it is a race that shows up under sustained 429 pressure. Roughly: deploy enough endpoints in one region that the mutation quota is exceeded for several minutes.

### [REQUIRED] Steps to reproduce

1. Deploy ~80+ 2nd-gen functions to a single region in one `firebase deploy --only functions`.
2. Let the Cloud Functions API return `429 Quota exceeded for quota metric 'Per project mutation requests' and limit 'Per project mutation requests per minute per region'` for some of them.
3. Observe the exit code and compare the deployed revisions against the source.

### [REQUIRED] Expected behavior

Either every endpoint deploys, or the command fails. `src/deploy/functions/release/index.ts` looks like it intends the latter — after `Fabricator::applyPlan` it calls `reporter.printErrors(summary)` and then:

```ts
const allErrors = summary.results.filter((r) => r.error).map((r) => r.error) as Error[];
if (allErrors.length) {
...
throw new FirebaseError("There was an error deploying functions", { ...opts, exit: 2 });
}
```

### [REQUIRED] Actual behavior

**The command exited 0 with two endpoints never updated, and printed no error summary.**

From the run's log:

- 81 endpoints logged `functions: updating Node.js 24 (2nd Gen) function (europe-west3)...`
- 79 logged `functions[(europe-west3)] Successful update operation.`
- One logged the 429 and then `⚠ functions: failed to update function projects//locations/europe-west3/functions/`, and nothing further
- One logged only `updating ...` and then nothing at all — no success, no error

And critically, **none** of these appeared:

- `Functions deploy had errors with the following functions:` (`reporter.printErrors`)
- `Deploy complete!`
- any artifact-cleanup output — `setupArtifactCleanupPolicies` is the statement right after `printTriggerUrls`, so execution seems not to have reached it either

The last line of the step was the final endpoint's success message, ~30 ms before the CI step ended. Exit code 0.

Verified against the Cloud Functions v2 API afterwards: both endpoints were still on their previous revision, and stayed there for three days across two further "successful" deploys of the same shape (the same bundle hash means they were retried and dropped the same way each time). Every other endpoint showed an `updateTime` inside the deploy window.

So it looks like the process can leave `applyPlan` — or exit outright — without `summary.results` carrying the failed operations, so the `allErrors` check finds nothing to throw and the CLI reports success. I have not been able to pin down the mechanism from the outside; `wrapOperation` does record `op.error` for non-retryable codes, so my guess is something in the throttler's retry-exhaustion path, but that is speculation.

### Impact

This is a silent partial deploy. CI reports green, and production keeps serving old code for the affected functions with nothing anywhere to indicate it. A non-zero exit — or even just the existing error summary — would have surfaced it immediately.

Happy to supply more of the raw log or test a patch.

Contributor guide

Open the contributing guide

Research direction

Start in src/deploy/functions/release/index.ts and trace Fabricator::applyPlan through reporter.printErrors and the allErrors check. Reproduce the quota-pressure deployment with roughly 80 functions, then inspect wrapOperation and the throttler retry-exhaustion path. Done means failed or incomplete endpoint updates are retained in the summary, printed, and cause a non-zero deploy exit.

Written by the indexing model from the issue text.

Assessment

Tech stack
node.js, typescript
Domain
cli
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.