firebase / firebase/firebase-tools
Killed deploy orphans the Python discovery server (serving.py); orphans wedge and later deploys hang forever on connect ETIMEDOUT
- Dominant language
- TypeScript
- Stars
- 4.5k
- Forks
- 1.3k
- Avg merge
- 1d 12h
- Merged PRs (30d)
- 84
Description
### Environment
- firebase-tools: 15.9.1 (repo-pinned devDependency, run from `node_modules/.bin/firebase`)
- Platform: macOS 15 (Apple Silicon), self-hosted GitHub Actions runners
- Project: multi-codebase functions — one nodejs codebase + two Python codebases (`firebase-functions` Python SDK)
- Command: `firebase deploy --only functions --force --project ` in CI, with a job-level 60-minute timeout
### Problem
When a `firebase deploy` run is killed externally mid-flight (CI job cancellation or timeout), the Python discovery admin server the CLI spawned (`firebase_functions/private/serving.py`, launched with `ADMIN_PORT=`) is left running. The orphan keeps its admin port bound but eventually stops accepting connections.
Once one or more such orphans exist on the host, **subsequent deploys hang indefinitely** during "Loading and analyzing source code for codebase <python-codebase>": every localhost call to the admin port — including `/__/quitquitquit` — fails with `connect ETIMEDOUT 127.0.0.1:` (timeout, not refused: a bound-but-not-accepting listener), and the CLI never errors out. In our CI this meant each affected deploy ran until the 60-minute job timeout killed it — which orphaned *its* discovery server too. Over 5 days ~35 orphaned `serving.py` processes accumulated on one runner host, at which point every deploy landing on it hung.
Debug log excerpt from a hung deploy (nodejs codebase discovery succeeded in ~1s just before this):
```
[debug] Running admin server with args: ["python3.12","/site-packages/firebase_functions/private/serving.py"] and env: {..., "ADMIN_PORT":"8120"} in /firebase/functions/python-ytmusicapi
[info] * Serving Flask app 'serving'
[debug] Failed to call quitquitquit. This often means the server failed to start request to http://127.0.0.1:8120/__/quitquitquit failed, reason: connect ETIMEDOUT 127.0.0.1:8120
[error] * Running on http://127.0.0.1:8120
```
(no further log lines; the process sat there until the CI job timeout ~59 minutes later)
### Verification
Killing the orphans (`pkill -f "firebase_functions/private/serving.py"` as the runner user) immediately restored ~2-minute deploys **on the same host** — reproduced across two hung runs and one clean rerun.
### Expected behavior
1. The CLI should hard-kill the discovery child process (SIGKILL by pid / process group) when `/__/quitquitquit` is unreachable, and on its own exit paths — an HTTP-only shutdown can never clean up a wedged server.
2. Discovery should fail fast with a clear error when the admin port cannot be reached within a bounded time, instead of hanging the whole deploy indefinitely.
### Workaround
A CI step before `firebase deploy` that pkills stale `serving.py` processes.
Contributor guide
Research direction
Start at the firebase deploy discovery flow and inspect how it launches and shuts down firebase_functions/private/serving.py, especially the /__/quitquitquit request. Reproduce a killed deployment with the Python codebase, then verify that an unreachable admin port produces a bounded, clear failure and that the discovery process is cleaned up on CLI exit.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, typescript
- Domain
- backend, cli
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100