firebase / firebase/firebase-tools

App Hosting rollout creation fails after five retries while new build is still becoming visible

Open
#11,078 0 comments 0 reactions 0 assignees View on GitHub
api: apphosting type: bug
Dominant language
TypeScript
Stars
4.5k
Forks
1.3k
Avg merge
1d 12h
Merged PRs (30d)
84

Description

### [REQUIRED] Environment info

**firebase-tools:** 15.28.2

**Platform:** GitHub Actions `ubuntu-latest`; invoked non-interactively via `bunx --bun firebase-tools@15.28.2`

The relevant retry logic is unchanged on the current `main` branch at the time of filing.

### [REQUIRED] Test case

Run `apphosting:rollouts:create` for an App Hosting backend connected to GitHub, selecting an exact commit:

```sh
firebase apphosting:rollouts:create BACKEND_ID \
--project PROJECT_ID \
--git-commit FULL_COMMIT_SHA \
--force \
--non-interactive
```

The race occurs when `createBuild` returns its long-running operation but the new Build resource is not yet visible to `CreateRollout(validateOnly=true)` within the CLI's five validation attempts.

### [REQUIRED] Steps to reproduce

1. Authenticate the CLI to a project containing a GitHub-connected App Hosting backend.
2. Run the command above for a valid commit.
3. Have the App Hosting control plane take longer than the CLI's short fixed retry window to make the newly created Build resource visible to rollout validation.
4. Observe that each rollout validation request returns HTTP 400 because the build is not found.
5. After the fifth attempt, the CLI exits 1 even though the build creation operation is still in progress.

The race is visible in `orchestrateRollout`: it starts `createBuild`, then immediately calls `createRollout(..., validateOnly=true)`. HTTP 400 is retried only five times with one-second sleeps. The CLI does not begin polling the build operation until after rollout creation succeeds:

- Released implementation: https://github.com/firebase/firebase-tools/blob/v15.28.2/src/apphosting/rollout.ts
- Current implementation: https://github.com/firebase/firebase-tools/blob/main/src/apphosting/rollout.ts

### [REQUIRED] Expected behavior

The CLI should treat temporary build nonexistence as normal operation progress. It should poll the build creation operation (or retry build visibility using the existing App Hosting operation-poller timeout/backoff policy) before validating and creating the rollout.

If the build eventually fails, the CLI should report the final build failure and build logs. If it succeeds, rollout creation should proceed.

### [REQUIRED] Actual behavior

The CLI gives the build only five validation attempts, separated by one-second sleeps, and then reports a terminal rollout failure:

```text
i You may also track this rollout at:
- Starting a new rollout; this may take a few minutes. It's safe to exit now.
✖ Rollout failed.

Error: Request to https://firebaseapphosting.googleapis.com/v1beta/projects/.../backends/.../rollouts?rolloutId=build-2026-09-11-001&validateOnly=true had HTTP Error: 400, The request was invalid: build "build-2026-09-11-001" was not found and is invalid
```

We worked around the race by calling the App Hosting API directly in this order:

1. Create the build.
2. Poll the returned build operation until complete.
3. Verify the Build resource and its final state.
4. Create the rollout.
5. Poll the rollout operation and verify production traffic.

That implementation no longer encounters the premature `build was not found` failure and has completed subsequent deployments successfully.

Contributor guide

Open the contributing guide

Research direction

Start in src/apphosting/rollout.ts, focusing on orchestrateRollout and the existing App Hosting operation-poller timeout/backoff policy. Run the reported apphosting:rollouts:create command against a GitHub-connected backend and observe build visibility during validation. Done means rollout validation waits for the build operation or visibility, successful builds proceed to rollout creation, and failed builds report their final failure and logs.

Written by the indexing model from the issue text.

Assessment

Tech stack
typescript
Domain
backend-api-design, cli, cloud
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
72/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.