firebase / firebase/firebase-tools

Killed deploy orphans the Python discovery server (serving.py); orphans wedge and later deploys hang forever on connect ETIMEDOUT

Open
#10,847 2 comments 0 reactions 0 assignees View on GitHub
api: functions reproducible type: bug
Dominant language
TypeScript
Stars
4.5k
Forks
1.3k
Avg merge
1d 12h
Merged PRs (30d)
84

Description

### Environment
- firebase-tools: 15.9.1 (repo-pinned devDependency, run from `node_modules/.bin/firebase`)
- Platform: macOS 15 (Apple Silicon), self-hosted GitHub Actions runners
- Project: multi-codebase functions — one nodejs codebase + two Python codebases (`firebase-functions` Python SDK)
- Command: `firebase deploy --only functions --force --project ` in CI, with a job-level 60-minute timeout

### Problem
When a `firebase deploy` run is killed externally mid-flight (CI job cancellation or timeout), the Python discovery admin server the CLI spawned (`firebase_functions/private/serving.py`, launched with `ADMIN_PORT=`) is left running. The orphan keeps its admin port bound but eventually stops accepting connections.

Once one or more such orphans exist on the host, **subsequent deploys hang indefinitely** during "Loading and analyzing source code for codebase <python-codebase>": every localhost call to the admin port — including `/__/quitquitquit` — fails with `connect ETIMEDOUT 127.0.0.1:` (timeout, not refused: a bound-but-not-accepting listener), and the CLI never errors out. In our CI this meant each affected deploy ran until the 60-minute job timeout killed it — which orphaned *its* discovery server too. Over 5 days ~35 orphaned `serving.py` processes accumulated on one runner host, at which point every deploy landing on it hung.

Debug log excerpt from a hung deploy (nodejs codebase discovery succeeded in ~1s just before this):

```
[debug] Running admin server with args: ["python3.12","/site-packages/firebase_functions/private/serving.py"] and env: {..., "ADMIN_PORT":"8120"} in /firebase/functions/python-ytmusicapi
[info] * Serving Flask app 'serving'
[debug] Failed to call quitquitquit. This often means the server failed to start request to http://127.0.0.1:8120/__/quitquitquit failed, reason: connect ETIMEDOUT 127.0.0.1:8120
[error] * Running on http://127.0.0.1:8120
```
(no further log lines; the process sat there until the CI job timeout ~59 minutes later)

### Verification
Killing the orphans (`pkill -f "firebase_functions/private/serving.py"` as the runner user) immediately restored ~2-minute deploys **on the same host** — reproduced across two hung runs and one clean rerun.

### Expected behavior
1. The CLI should hard-kill the discovery child process (SIGKILL by pid / process group) when `/__/quitquitquit` is unreachable, and on its own exit paths — an HTTP-only shutdown can never clean up a wedged server.
2. Discovery should fail fast with a clear error when the admin port cannot be reached within a bounded time, instead of hanging the whole deploy indefinitely.

### Workaround
A CI step before `firebase deploy` that pkills stale `serving.py` processes.

Contributor guide

Open the contributing guide

Research direction

Start at the firebase deploy discovery flow and inspect how it launches and shuts down firebase_functions/private/serving.py, especially the /__/quitquitquit request. Reproduce a killed deployment with the Python codebase, then verify that an unreachable admin port produces a bounded, clear failure and that the discovery process is cleaned up on CLI exit.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, typescript
Domain
backend, cli
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.