firebase / firebase/firebase-tools
Gen 2 function deploys can exhaust regional CPU/memory because release concurrency is fixed at 40
- Dominant language
- TypeScript
- Stars
- 4.5k
- Forks
- 1.3k
- Avg merge
- 1d 12h
- Merged PRs (30d)
- 84
Description
### [REQUIRED] Environment info
**firebase-tools:** 15.25.1. I also checked 15.28.2, the latest release at the time of filing, and the relevant concurrency remains unchanged.
**Platform:** Ubuntu (GitHub Actions), Node.js 22
### [REQUIRED] Test case
A Firebase project with approximately 70 Gen 2 functions in one region. Many functions share a bundle, so a shared change causes most of them to require new revisions in one deploy.
The project's regional Cloud Run quota is 20 vCPU / 40 GiB.
### [REQUIRED] Steps to reproduce
1. Change shared function code so many Gen 2 functions need an update.
2. Run:
```sh
firebase deploy --only functions --project --non-interactive --force
```
3. Observe many Gen 2 update operations being released concurrently.
The Gen 2 update path uses `functionExecutor`, while the release executor has a fixed concurrency of 40:
- https://github.com/firebase/firebase-tools/blob/ec3269200e9141e1b9e9ad19f70c1ae376fba065/src/deploy/functions/release/index.ts#L80-L104
- https://github.com/firebase/firebase-tools/blob/ec3269200e9141e1b9e9ad19f70c1ae376fba065/src/deploy/functions/release/fabricator.ts#L760-L785
`firebase deploy --help` does not expose a supported way to lower this concurrency.
### [REQUIRED] Expected behavior
A bulk functions deploy should schedule Gen 2 updates so temporary revision health checks do not exhaust the project's regional CPU/memory quota.
At minimum, the CLI should expose a supported concurrency setting so CI can choose a value appropriate for the project's quota and function CPU/memory sizes. A quota-aware default would be even better.
### [REQUIRED] Actual behavior
One deploy produced 56 errors like these, many within the same second:
```text
Could not create or update Cloud Run service , Container Healthcheck failed.
Quota exceeded for total allowable CPU per project per region.
```
The deployment failed even though the services themselves were healthy and deployable individually.
Workarounds tried:
- CPU-aware batches still failed because old/new revision overlap and deployment health checks consume regional quota.
- Fixed 15-minute waits made the deployment take more than 90 minutes and still eventually hit the memory quota.
- Pre-updating existing Cloud Run services with `gcloud run services update --no-deploy-health-check` allowed the same bulk Firebase deployment to complete, but this removes eager startup validation and cannot prepare a service that does not exist yet.
This is specifically about aggregate CPU/memory consumed by concurrent Gen 2 revision deployment health checks. It is separate from Cloud Functions API write-rate quota errors and from a function's configured `maxInstances` being individually too high.
Related older reports cover retries/rate limits or the CLI exit status, but not a supported limit for this Gen 2 release concurrency:
- https://github.com/firebase/firebase-tools/issues/2606
- https://github.com/firebase/firebase-tools/issues/7949
Contributor guide
Research direction
Start in src/deploy/functions/release/index.ts and src/deploy/functions/release/fabricator.ts, where the issue identifies the fixed release concurrency and the Gen 2 update path. Trace how deploy options reach the release executor, then reproduce with the linked bulk-deploy command; done means a supported concurrency setting can prevent regional quota exhaustion during Gen 2 deployments.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- nodejs, typescript
- Domain
- cli, cloud, devops
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100