Lightning-AI / Lightning-AI/sdk
Python REST client can block indefinitely in TLS handshake when `request_timeout` is omitted
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 10
- Forks
- 3
- Avg merge
- 2d 1h
- Merged PRs (30d)
- 32
Description
## Summary
Lightning SDK's generated Python REST client passes an explicit `timeout=None` to urllib3 whenever `_request_timeout` is omitted.
This allows a connection to block indefinitely during the TCP connection, TLS handshake, or response read. It also defeats `socket.setdefaulttimeout`, because urllib3 applies the explicit `None` to the newly created socket.
We have observed production jobs remain blocked in `ssl.do_handshake` for many hours through this call path:
litlogger.log_metrics
-> Experiment.__getitem__
-> ExperimentStateSupport.resolve_remote_model
-> Teamspace.list_models
-> projects_service_get_project
-> RESTClientObject.request
-> urllib3
-> ssl.do_handshake
The behavior is reproducible entirely on localhost without Lightning credentials or access to the Lightning service.
## Environment
- Python: 3.12.12
- lightning-sdk: 2026.8.18
- urllib3: 2.5.0
- macOS arm64
The same behavior was previously observed with:
- lightning-sdk 2026.6.25.post0
- lightning-sdk 2026.7.9.post0
- urllib3 2.5.0
## Cause
`RESTClientObject.request()` converts an omitted `_request_timeout` into `timeout = None` ([source](https://github.com/Lightning-AI/sdk/blob/v2026.08.18/python/lightning_sdk/lightning_cloud/openapi/rest.py#L151-L158)) and then explicitly passes that value to `urllib3` ([source](https://github.com/Lightning-AI/sdk/blob/v2026.08.18/python/lightning_sdk/lightning_cloud/openapi/rest.py#L215-L220)).
When `_request_timeout` is omitted, the generated API layers propagate `None` all the way down.
urllib3 distinguishes between an omitted timeout and an explicit `None`. For an explicit value, including `None`, `create_connection` calls:
``` python
sock.settimeout(timeout)
```
Consequently, `timeout=None` puts the fresh socket into blocking mode and overrides any process-wide `socket.setdefaulttimeout(...)`.
## Reproducer
```python
from __future__ import annotations
import logging
import socket
import threading
import time
import lightning_sdk
from lightning_sdk.lightning_cloud.openapi import Configuration
from lightning_sdk.lightning_cloud.openapi.rest import RESTClientObject
logging.getLogger("urllib3").setLevel(logging.CRITICAL)
def run_case(
name: str,
request_timeout: object,
socket_default: float | None,
) -> None:
listener = socket.socket()
listener.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
listener.bind(("127.0.0.1", 0))
listener.listen()
listener.settimeout(0.05)
stop = threading.Event()
connections: list[socket.socket] = []
def serve() -> None:
while not stop.is_set():
try:
connection, _ = listener.accept()
except TimeoutError:
continue
except OSError:
return
# Accept TCP, but never send a TLS ServerHello.
connections.append(connection)
threading.Thread(target=serve, daemon=True).start()
socket.setdefaulttimeout(socket_default)
configuration = Configuration()
configuration.verify_ssl = False
rest = RESTClientObject(configuration)
outcome: list[str] = []
started = time.monotonic()
def request() -> None:
try:
rest.GET(
f"https://127.0.0.1:{listener.getsockname()[1]}/wedge",
_request_timeout=request_timeout,
)
outcome.append("completed")
except BaseException as exc:
outcome.append(type(exc).__name__)
worker = threading.Thread(target=request, daemon=True)
worker.start()
worker.join(1.5)
if worker.is_alive():
print(f"{name}: WEDGED past 1.5s; accepted={len(connections)}")
else:
elapsed = time.monotonic() - started
print(
f"{name}: raised {outcome[0]} after {elapsed:.3f}s; "
f"accepted={len(connections)}"
)
stop.set()
listener.close()
for connection in connections:
connection.close()
worker.join(0.5)
socket.setdefaulttimeout(None)
print("lightning_sdk", lightning_sdk.__version__)
run_case("default", None, None)
run_case("socket.setdefaulttimeout(0.2)", None, 0.2)
run_case(
"explicit _request_timeout=(0.2, 0.2)",
(0.2, 0.2),
None,
)
```
Run with:
```bash
uv run --isolated --no-project \
--with 'lightning-sdk==2026.8.18' \
python reproduce_timeout.py
```
Observed:
```text
lightning_sdk 2026.8.18
default: WEDGED past 1.5s; accepted=1
socket.setdefaulttimeout(0.2): WEDGED past 1.5s; accepted=1
explicit _request_timeout=(0.2, 0.2):
raised MaxRetryError after 0.828s; accepted=4
```
The blocked worker stack ends in:
```text
ssl.py:1319 in do_handshake
urllib3/util/ssl_.py in _ssl_wrap_socket_impl
urllib3/connection.py in connect
urllib3/connectionpool.py in _validate_conn
urllib3/poolmanager.py in urlopen
lightning_sdk/lightning_cloud/openapi/rest.py in request
```
## Expected behavior
Lightning SDK network operations should have a finite, configurable default connect/read timeout when callers do not provide `_request_timeout`.
Explicit per-request timeouts should continue to override that default.
At minimum, when no SDK timeout is configured, the REST client should avoid passing an explicit `timeout=None` so that urllib3 does not override the process socket default.
## Additional retry consideration
The timeout currently applies to one urllib3 attempt, not one logical SDK operation. urllib3 defaults to three retries, and higher-level Lightning clients may apply another retry loop.
With a 30-second timeout, a persistently silent endpoint can therefore take approximately:
4 urllib3 attempts * 7 SDK attempts * 30 seconds
+ 31.5 seconds of SDK backoff
≈ 871.5 seconds
It would be helpful for the SDK to define the timeout and retry policy together, or expose a logical-call deadline, to avoid multiplying two independent retry layers.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with lightning_cloud/openapi/rest.py at the cited timeout conversion and urllib3 call sites, then run the localhost reproducer without Lightning credentials. Done means omitted _request_timeout no longer permits an indefinite handshake while explicit per-request timeouts still override the default; the retry interaction may need clarification.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- api, networking
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 68/100