firebase / firebase/firebase-tools

Gen 2 function deploys can exhaust regional CPU/memory because release concurrency is fixed at 40

Open
#11,001 3 comments 0 reactions 0 assignees View on GitHub
api: functions Needs: Author Feedback no-recent-activity reproducible type: bug
Dominant language
TypeScript
Stars
4.5k
Forks
1.3k
Avg merge
1d 12h
Merged PRs (30d)
84

Description

### [REQUIRED] Environment info

**firebase-tools:** 15.25.1. I also checked 15.28.2, the latest release at the time of filing, and the relevant concurrency remains unchanged.

**Platform:** Ubuntu (GitHub Actions), Node.js 22

### [REQUIRED] Test case

A Firebase project with approximately 70 Gen 2 functions in one region. Many functions share a bundle, so a shared change causes most of them to require new revisions in one deploy.

The project's regional Cloud Run quota is 20 vCPU / 40 GiB.

### [REQUIRED] Steps to reproduce

1. Change shared function code so many Gen 2 functions need an update.
2. Run:

```sh
firebase deploy --only functions --project --non-interactive --force
```

3. Observe many Gen 2 update operations being released concurrently.

The Gen 2 update path uses `functionExecutor`, while the release executor has a fixed concurrency of 40:

- https://github.com/firebase/firebase-tools/blob/ec3269200e9141e1b9e9ad19f70c1ae376fba065/src/deploy/functions/release/index.ts#L80-L104
- https://github.com/firebase/firebase-tools/blob/ec3269200e9141e1b9e9ad19f70c1ae376fba065/src/deploy/functions/release/fabricator.ts#L760-L785

`firebase deploy --help` does not expose a supported way to lower this concurrency.

### [REQUIRED] Expected behavior

A bulk functions deploy should schedule Gen 2 updates so temporary revision health checks do not exhaust the project's regional CPU/memory quota.

At minimum, the CLI should expose a supported concurrency setting so CI can choose a value appropriate for the project's quota and function CPU/memory sizes. A quota-aware default would be even better.

### [REQUIRED] Actual behavior

One deploy produced 56 errors like these, many within the same second:

```text
Could not create or update Cloud Run service , Container Healthcheck failed.
Quota exceeded for total allowable CPU per project per region.
```

The deployment failed even though the services themselves were healthy and deployable individually.

Workarounds tried:

- CPU-aware batches still failed because old/new revision overlap and deployment health checks consume regional quota.
- Fixed 15-minute waits made the deployment take more than 90 minutes and still eventually hit the memory quota.
- Pre-updating existing Cloud Run services with `gcloud run services update --no-deploy-health-check` allowed the same bulk Firebase deployment to complete, but this removes eager startup validation and cannot prepare a service that does not exist yet.

This is specifically about aggregate CPU/memory consumed by concurrent Gen 2 revision deployment health checks. It is separate from Cloud Functions API write-rate quota errors and from a function's configured `maxInstances` being individually too high.

Related older reports cover retries/rate limits or the CLI exit status, but not a supported limit for this Gen 2 release concurrency:

- https://github.com/firebase/firebase-tools/issues/2606
- https://github.com/firebase/firebase-tools/issues/7949

Contributor guide

Open the contributing guide

Research direction

Start in src/deploy/functions/release/index.ts and src/deploy/functions/release/fabricator.ts, where the issue identifies the fixed release concurrency and the Gen 2 update path. Trace how deploy options reach the release executor, then reproduce with the linked bulk-deploy command; done means a supported concurrency setting can prevent regional quota exhaustion during Gen 2 deployments.

Written by the indexing model from the issue text.

Assessment

Tech stack
nodejs, typescript
Domain
cli, cloud, devops
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.