dask / dask/dask-gateway

cleanup_lingering_clusters atexit fails to stop cluster: RuntimeError in DNS resolution after executor shutdown

Open
#930 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
148
Forks
93
PR merge metrics
No merged PRs in 30d

Description

**Describe the issue**:

`GatewayCluster`'s atexit handler `cleanup_lingering_clusters` fails to send the HTTP DELETE to the gateway server during interpreter shutdown. The cluster **keeps running and billing continues** until it times out or is manually stopped.

The root cause: CPython's `concurrent.futures.thread._python_exit()` atexit handler runs **before** dask-gateway's `cleanup_lingering_clusters` and kills all `ThreadPoolExecutor` instances while also setting a module-level `_shutdown = True` flag that blocks even new executors. When `_stop_internal` → `Gateway._request()` → `aiohttp.ClientSession.request()` tries to resolve DNS via aiohttp's default `ThreadedResolver`, it calls `loop.run_in_executor(None, socket.getaddrinfo, ...)` on a dead executor → `RuntimeError` → DELETE never sent.

**Minimal Complete Verifiable Example**:

```python
# Requires a running dask-gateway server — cannot be reproduced with LocalCluster.
from dask_gateway import GatewayCluster

if __name__ == "__main__":
cluster = GatewayCluster(address="https://your-gateway.example.com")
client = cluster.get_client()

future = client.submit(lambda: 42)
print(future.result())
# Exit without cluster.close() — Ctrl+C or natural script end.
# Expected: cleanup_lingering_clusters atexit sends DELETE, cluster stops.
# Actual: RuntimeError in DNS resolution, DELETE never sent, cluster keeps running.
```

**Anything else we need to know?**:

The call chain is: `cleanup_lingering_clusters` (atexit) → `GatewayCluster.close()` → `_stop_internal` → `Gateway._stop_cluster(name)` → `Gateway._request("DELETE", url)` → `aiohttp.ClientSession.request()` → `ThreadedResolver.resolve()` → `loop.getaddrinfo()` → `loop.run_in_executor(None, ...)` → `RuntimeError`.

Creating a **new** `ThreadPoolExecutor` does not help — CPython's `concurrent.futures.thread.submit()` checks both `self._shutdown` (instance flag) and the module-level `_shutdown` flag set by `_python_exit()`.

Suggested fix: when `_stop_cluster` raises `RuntimeError` during shutdown, retry with a throwaway `aiohttp.ClientSession` using a synchronous DNS resolver that calls `socket.getaddrinfo()` directly (no executor needed). Blocking the event loop is acceptable during shutdown.

```python
import socket

class _SyncResolver:
"""Blocking DNS resolver — fallback for shutdown when executors are dead."""
async def resolve(self, host, port=0, family=socket.AF_INET):
infos = socket.getaddrinfo(host, port, family, socket.SOCK_STREAM)
if not infos:
raise OSError(f"DNS lookup failed for {host}")
return [
{"hostname": host, "host": a[-1][0], "port": a[-1][1],
"family": a[0], "proto": a[2],
"flags": socket.AI_NUMERICHOST | socket.AI_NUMERICSERV}
for a in infos
]

async def close(self):
pass
```

In `_stop_internal`:
```python
try:
await self.gateway._stop_cluster(self.name)
except RuntimeError:
# Executors dead at shutdown — retry with blocking DNS
connector = aiohttp.TCPConnector(resolver=_SyncResolver())
async with aiohttp.ClientSession(
connector=connector, cookie_jar=aiohttp.CookieJar(unsafe=True)
) as session:
url = f"{self.gateway.address}/api/v1/clusters/{self.name}"
resp = await session.request("DELETE", url, **self.gateway._request_kwargs)
if resp.status == 401:
hdrs, _ctx = self.gateway.auth.pre_request(resp)
await session.request("DELETE", url, headers=hdrs, **self.gateway._request_kwargs)
```

Related: `dask/distributed` issue — `done_callback` raises the same `RuntimeError` during shutdown (separate bug, same root cause).

**Environment**:

- Dask version: 2025.9.1
- dask-gateway version: 2025.4.0
- aiohttp version: 3.13.3
- Python version: 3.11.14
- Operating System: macOS 15.7.4 (arm64)
- Install method: uv (pip)

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.