ApeWorX / ApeWorX/web3.py

AsyncHTTPProvider requests can deadlock HTTPSessionManager._lock when cancelled

Open
#3,867 4 comments 1 reaction 0 assignees View on GitHub
Dominant language
Python
Stars
5.5k
Forks
1.7k
Avg merge
3d 10h
Merged PRs (30d)
2

Description

### What happened?

I run a market-making system that talks to 3 independent RPC vendors through separate `AsyncHTTPProvider` instances, wrapping calls in `asyncio.wait_for(...)`. Roughly once a day, **all HTTP requests on all providers started timing out simultaneously and never recovered** — `eth_sendRawTransaction` and `eth_call` alike, across unrelated vendors — while WebSocket providers in the same process kept working. Only a process restart cleared it. Process-level TCP monitoring showed no connection churn or SYN backlog; this pointed away from DNS/connectivity exhaustion and toward an in-process stall.

Root cause, two compounding defects:

1. `HTTPSessionManager._lock` is a **class attribute** (`web3/_utils/http_session_manager.py`), so every provider instance in the process shares one `threading.Lock`, and `async_cache_and_return_session()` takes it on **every request**, including cache hits. In web3.py 7.x, the async path (unlike the sync one) acquires the lock before checking whether the session is already present in `self.session_cache`, so the common cached-session path still pays the `run_in_executor(lock.acquire)` cost and is exposed to cancellation at the lock-acquire boundary.
2. `async_lock()` (`web3/_utils/async_caching.py`) acquires that lock via `loop.run_in_executor(thread_pool, lock.acquire)` and is **not cancellation-safe**:
- If the awaiting task is cancelled (e.g. `asyncio.wait_for` timeout) while the executor thread has already started and is blocked inside `lock.acquire()`, the thread cannot be cancelled. It later acquires the lock — and **nothing ever releases it**. From then on, every request in the process blocks forever on the dead lock.
- Additionally, the cleanup `finally: if lock.locked(): lock.release()` releases the lock **whoever holds it** — a cancelled waiter can release a lock currently held by a different coroutine, breaking mutual exclusion.

Why this got much worse in 7.x: in 6.x the equivalent code used one module-level `ThreadPoolExecutor(max_workers=1)`, so at most one `acquire` could be started at any time (queued ones cancel cleanly) — the vulnerable window was practically zero. In 7.x each `AsyncHTTPProvider` gets its own `HTTPSessionManager` with a 5-worker `session_pool`, while the lock stayed globally shared via the class attribute. Under concurrency, many acquires are simultaneously started-and-blocked on the same lock, so an ordinary client-side timeout is very likely to cancel one of them in the vulnerable window.

Reproduced on web3 7.16.0. The repro below uses only public API (`make_request` + `asyncio.wait_for`) against a local aiohttp server and wedges the process on the first round in nearly every run.

### Code that produced the error

```python
"""Cancelling AsyncHTTPProvider requests permanently deadlocks the class-level
HTTPSessionManager._lock, wedging every AsyncHTTPProvider in the process."""

import asyncio
import os
import threading

from aiohttp import web

from web3 import AsyncWeb3
from web3._utils.http_session_manager import HTTPSessionManager

PORT = 18545
URL = f"http://127.0.0.1:{PORT}"

async def rpc_handler(request: web.Request) -> web.Response:
body = await request.json()
await asyncio.sleep(0.05) # simulate a slow-ish RPC node
return web.json_response({"jsonrpc": "2.0", "id": body["id"], "result": "0x1"})

async def start_server() -> web.AppRunner:
app = web.Application()
app.router.add_post("/", rpc_handler)
runner = web.AppRunner(app)
await runner.setup()
await web.TCPSite(runner, "127.0.0.1", PORT).start()
return runner

async def one_request(provider, i: int) -> None:
# Ordinary client-side timeout handling, as any production caller does.
try:
await asyncio.wait_for(
provider.make_request("eth_blockNumber", []),
timeout=0.001 + (i % 30) * 0.002,
)
except asyncio.TimeoutError:
pass

async def main() -> None:
runner = await start_server()
# Three independent providers, e.g. three RPC vendors for redundancy.
providers = [AsyncWeb3.AsyncHTTPProvider(URL) for _ in range(3)]

for round_no in range(1, 21):
await asyncio.gather(*(one_request(p, i) for i, p in enumerate(providers * 30)))
await asyncio.sleep(0.5) # let stragglers finish; lock should be free now
if HTTPSessionManager._lock.locked():
print(
f"round {round_no}: class-level HTTPSessionManager._lock "
f"is still LOCKED after all caller coroutines have returned or timed out"
)
break
else:
print("not reproduced in 20 rounds (timing dependent, run again)")
await runner.cleanup()
return

# The whole process is now wedged: even a brand-new provider instance
# (its own HTTPSessionManager, but the SAME class-level _lock) hangs.
fresh = AsyncWeb3.AsyncHTTPProvider(URL)
try:
await asyncio.wait_for(fresh.make_request("eth_blockNumber", []), timeout=5)
print("fresh provider request succeeded (not wedged)")
except asyncio.TimeoutError:
print(
"fresh provider request timed out after 5s "
"-> every AsyncHTTPProvider in this process is now unusable"
)
print(
f"lock still locked: {HTTPSessionManager._lock.locked()}, "
f"threads alive: {threading.active_count()}"
)
await runner.cleanup()
# Without this, interpreter shutdown hangs forever: atexit joins the
# executor thread that is still blocked inside lock.acquire().
os._exit(0)

asyncio.run(main())
```

### Full error output

```shell
$ python repro.py
round 1: class-level HTTPSessionManager._lock is still LOCKED after all caller coroutines have returned or timed out
fresh provider request timed out after 5s -> every AsyncHTTPProvider in this process is now unusable
lock still locked: True, threads alive: 18
```

### Fill this section in if you know how this could or should be fixed

1. **Preferred: drop the thread dance for the async path entirely.** The async session cache key is already scoped by `id(asyncio.get_event_loop())`, so a per-loop `asyncio.Lock` is sufficient to guard the cache. That removes `run_in_executor`, is inherently cancellation-safe, and avoids burning a 5-thread pool per provider. A lock-free fast path for cache hits (the common case — currently every request pays the executor round-trip even on hits) would also help latency.
2. **Make `_lock` an instance attribute regardless.** `_lock: threading.Lock = threading.Lock()` on the class means one poisoned/contended manager wedges every provider in the process; instance scope at least contains the blast radius.
3. **If a `threading.Lock` must remain, `async_lock()` needs cancellation-safe ownership tracking.** It should release the lock only if the current coroutine actually acquired it. If cancellation happens after the executor task has started but before the awaiter resumes, the orphaned acquire should arrange to release the lock once it eventually succeeds.

Pseudo-code sketch:

```
fut = loop.run_in_executor(thread_pool, lock.acquire)
acquired = False
try:
await asyncio.shield(fut)
acquired = True
yield
except asyncio.CancelledError:
if not acquired:
fut.add_done_callback(release_if_acquired_later)
raise
finally:
if acquired:
lock.release()
```

That said, the cleaner fix for the async path is likely to avoid `threading.Lock` and `run_in_executor` entirely, and use a per-event-loop `asyncio.Lock` around the async session cache instead.

### web3 Version

7.16.0

### Python Version

3.11.15

### Operating System

linux

### Output from `pip freeze`

```shell
aiodns 4.0.4
aiofiles 24.1.0
aiohappyeyeballs 2.7.1
aiohttp 3.14.1
aiohttp-session 2.12.0
aiosignal 1.4.0
annotated-types 0.7.0
APScheduler 3.10.4
attrs 26.1.0
bitarray 3.8.2
boto3 1.43.40
botocore 1.43.40
ccxt 4.5.36
certifi 2026.6.17
cffi 2.0.0
charset-normalizer 3.4.8
ckzg 2.1.7
coincurve 21.0.0
cryptography 42.0.8
cytoolz 1.1.0
eth_abi 5.2.0
eth-account 0.13.7
eth-hash 0.8.0
eth-keyfile 0.8.1
eth-keys 0.7.0
eth-rlp 2.2.0
eth-typing 6.0.0
eth-utils 6.0.0
frozenlist 1.8.0
hexbytes 1.3.1
idna 3.18
jmespath 1.1.0
msgpack 1.2.1
multidict 6.7.1
nest-asyncio 1.6.0
numpy 1.26.4
packaging 26.0
pandas 1.5.3
parsimonious 0.10.0
pip 26.0.1
propcache 0.5.2
protobuf 5.29.3
psutil 6.0.0
pycares 5.0.1
pycparser 3.0
pycryptodome 3.23.0
pydantic 2.13.4
pydantic_core 2.46.4
python-dateutil 2.9.0.post0
pytz 2026.2
pyunormalize 17.0.0
PyYAML 6.0
regex 2026.6.28
requests 2.34.2
rlp 4.1.0
s3transfer 0.19.0
scipy 1.17.1
setuptools 83.0.0
six 1.17.0
sortedcontainers 2.4.0
toolz 1.1.0
types-requests 2.33.0.20260518
typing_extensions 4.16.0
typing-inspection 0.4.2
tzlocal 5.4.4
urllib3 2.7.0
uvloop 0.21.0
web3 7.16.0
websockets 15.0.1
wheel 0.46.3
yarl 1.24.2
```

Contributor guide

Open the contributing guide

Research direction

Start by reading web3/_utils/http_session_manager.py and web3/_utils/async_caching.py, then run the supplied cancellation reproducer against web3 7.16.0. Trace async_lock() and async_cache_and_return_session() through cancellation and verify that timed-out requests cannot leave the session lock held or wedge new AsyncHTTPProvider requests.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
api, backend
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
50/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.