Task subprocess can deadlock permanently when Linux fork() inherits a held OpenSSL lock from the supervisor
- Dominant language
- Python
- Stars
- 46.9k
- Forks
- 17.8k
- Avg merge
- 2d 9h
- Merged PRs (30d)
- 472
Description
Tasks that make outbound HTTPS calls (cloud-provider hooks/operators built on `google-auth`/`urllib3`/`requests` — e.g. `GKEStartPodOperator` or a Google-provider sensor) can hang indefinitely with no further progress. `execution_timeout` does not recover the task (thats a separate, related problem tracked in #53337).
`py-spy` dumps of a hung worker, taken minutes apart, confirm that every active thread in the process the task's own thread, plus unrelated background threads are frozen at the identical point, each building a fresh `SSLContext` for a *different* remote host:
```
__new__ (ssl.py:440)
create_urllib3_context (urllib3/util/ssl_.py:252)
_ssl_wrap_socket_and_match_hostname (urllib3/connection.py:928)
connect (urllib3/connection.py:812)
...
```
That rules out a network-specific hang and `SSLContext` construction itself is blocked process-wide.
The task subprocess is created via a bare `os.fork()` in `ActivitySubprocess.start()` (`task-sdk/src/airflow/sdk/execution_time/supervisor.py`), since `_should_use_exec()` only returns `True` on `darwin`:
```python
_FORK_EXEC_PLATFORMS = {"darwin"}
def _should_use_exec() -> bool:
return sys.platform in _FORK_EXEC_PLATFORMS
```
The supervisor process is multithreaded by the time any task starts (OTel exporter threads, per-connection `google-auth` background refresh threads, an OpenLineage listener, etc. are normal background threads in a long-running worker). `os.fork()` copies the address space. Still, only the calling thread survives in the child. If any sibling thread was midway through OpenSSL's global lock (taken inside `SSL_CTX_new()`, which every `ssl.SSLContext()` call goes through) at the exact instant of the fork, the child inherits that lock already held, permanently & the thread that held it no longer exists to release it. Every later `SSLContext` construction in that child, from any thread, then blocks forever.
### Steps to reproduce (a timing race, won't reproduce every attempt):
1. Run a task via the Task SDK on Linux, using a hook/operator that opens outbound HTTPS connections.
2. Ensure the supervisor has several concurrent background threads also doing HTTPS/TLS work at task-launch time (OTel exporter, one or more `google-auth` `Credentials` objects with regional-access-boundary refresh threads, OpenLineage listener).
3. Launch tasks in volume. Intermittently, a task hangs forever with no log output past its last HTTPS-triggering line.
4. `py-spy dump --pid --subprocesses` while hung, every active thread shows the same `ssl.py:440` frame.
### What do you think should happen instead?
The task should complete or eventually be killed and retried — not hang indefinitely due to C-library lock state inherited from an unrelated background thread.
Same hazard class already fixed for macOS (#64874, #65691 — Objective-C runtime corruption after bare `fork()`) and hit again independently on Linux (#65942/#65943 — Edge Worker forking a 22-thread process, corrupting Python's own import-lock state). Here the poisoned lock is OpenSSL's, and the affected fork is the Task SDK's own `ActivitySubprocess.start()`, not an executor-level fork — so the existing `_should_use_exec()` gate doesn't cover it, since the original macOS fix's reasoning ("Linux's resolver has no ObjC dependency") is specific to that hazard and doesn't generalize to OpenSSL's own fork-unsafe locking.
Following the Edge Worker precedent (#65943), a plausible fix shape is an opt-in "fresh interpreter instead of bare fork" setting for `ActivitySubprocess.start()` on Linux, defaulting to current behavior to avoid a blanket performance regression — open question for maintainers whether that reuses `execute_tasks_new_python_interpreter` or needs a dedicated setting.
### More info
- Related but distinct: #53337 (moving `execution_timeout` enforcement to the supervisor) would let a hang like this get killed and retried automatically — valuable, but doesn't fix the hang itself.
- Prior art for this hazard class: #64874, #65691 (macOS/ObjC), #65942/#65943 (Linux/Edge Worker, Python import locks). CPython itself emits `DeprecationWarning: This process is multi-threaded, use of fork() may lead to deadlocks in the child` in this exact scenario.
Contributor guide
Research direction
Start in task-sdk/src/airflow/sdk/execution_time/supervisor.py at ActivitySubprocess.start() and _should_use_exec(), then compare the Linux fork behavior with the prior macOS and Edge Worker fixes cited in the issue. Reproduce the timing race with concurrent HTTPS work and verify that task launches no longer inherit a permanently held OpenSSL lock, allowing completion or eventual kill and retry.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- backend
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100