Lightning-AI / Lightning-AI/sdk

Python REST client can block indefinitely in TLS handshake when `request_timeout` is omitted

Open
#132 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
10
Forks
3
Avg merge
2d 1h
Merged PRs (30d)
32

Description

## Summary

Lightning SDK's generated Python REST client passes an explicit `timeout=None` to urllib3 whenever `_request_timeout` is omitted.

This allows a connection to block indefinitely during the TCP connection, TLS handshake, or response read. It also defeats `socket.setdefaulttimeout`, because urllib3 applies the explicit `None` to the newly created socket.

We have observed production jobs remain blocked in `ssl.do_handshake` for many hours through this call path:

litlogger.log_metrics
-> Experiment.__getitem__
-> ExperimentStateSupport.resolve_remote_model
-> Teamspace.list_models
-> projects_service_get_project
-> RESTClientObject.request
-> urllib3
-> ssl.do_handshake

The behavior is reproducible entirely on localhost without Lightning credentials or access to the Lightning service.

## Environment

- Python: 3.12.12
- lightning-sdk: 2026.8.18
- urllib3: 2.5.0
- macOS arm64

The same behavior was previously observed with:

- lightning-sdk 2026.6.25.post0
- lightning-sdk 2026.7.9.post0
- urllib3 2.5.0

## Cause

`RESTClientObject.request()` converts an omitted `_request_timeout` into `timeout = None` ([source](https://github.com/Lightning-AI/sdk/blob/v2026.08.18/python/lightning_sdk/lightning_cloud/openapi/rest.py#L151-L158)) and then explicitly passes that value to `urllib3` ([source](https://github.com/Lightning-AI/sdk/blob/v2026.08.18/python/lightning_sdk/lightning_cloud/openapi/rest.py#L215-L220)).

When `_request_timeout` is omitted, the generated API layers propagate `None` all the way down.

urllib3 distinguishes between an omitted timeout and an explicit `None`. For an explicit value, including `None`, `create_connection` calls:

``` python
sock.settimeout(timeout)
```

Consequently, `timeout=None` puts the fresh socket into blocking mode and overrides any process-wide `socket.setdefaulttimeout(...)`.

## Reproducer

```python
from __future__ import annotations

import logging
import socket
import threading
import time

import lightning_sdk
from lightning_sdk.lightning_cloud.openapi import Configuration
from lightning_sdk.lightning_cloud.openapi.rest import RESTClientObject

logging.getLogger("urllib3").setLevel(logging.CRITICAL)

def run_case(
name: str,
request_timeout: object,
socket_default: float | None,
) -> None:
listener = socket.socket()
listener.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
listener.bind(("127.0.0.1", 0))
listener.listen()
listener.settimeout(0.05)

stop = threading.Event()
connections: list[socket.socket] = []

def serve() -> None:
while not stop.is_set():
try:
connection, _ = listener.accept()
except TimeoutError:
continue
except OSError:
return

# Accept TCP, but never send a TLS ServerHello.
connections.append(connection)

threading.Thread(target=serve, daemon=True).start()
socket.setdefaulttimeout(socket_default)

configuration = Configuration()
configuration.verify_ssl = False
rest = RESTClientObject(configuration)

outcome: list[str] = []
started = time.monotonic()

def request() -> None:
try:
rest.GET(
f"https://127.0.0.1:{listener.getsockname()[1]}/wedge",
_request_timeout=request_timeout,
)
outcome.append("completed")
except BaseException as exc:
outcome.append(type(exc).__name__)

worker = threading.Thread(target=request, daemon=True)
worker.start()
worker.join(1.5)

if worker.is_alive():
print(f"{name}: WEDGED past 1.5s; accepted={len(connections)}")
else:
elapsed = time.monotonic() - started
print(
f"{name}: raised {outcome[0]} after {elapsed:.3f}s; "
f"accepted={len(connections)}"
)

stop.set()
listener.close()
for connection in connections:
connection.close()
worker.join(0.5)
socket.setdefaulttimeout(None)

print("lightning_sdk", lightning_sdk.__version__)
run_case("default", None, None)
run_case("socket.setdefaulttimeout(0.2)", None, 0.2)
run_case(
"explicit _request_timeout=(0.2, 0.2)",
(0.2, 0.2),
None,
)
```

Run with:

```bash
uv run --isolated --no-project \
--with 'lightning-sdk==2026.8.18' \
python reproduce_timeout.py
```

Observed:

```text
lightning_sdk 2026.8.18
default: WEDGED past 1.5s; accepted=1
socket.setdefaulttimeout(0.2): WEDGED past 1.5s; accepted=1
explicit _request_timeout=(0.2, 0.2):
raised MaxRetryError after 0.828s; accepted=4
```

The blocked worker stack ends in:

```text
ssl.py:1319 in do_handshake
urllib3/util/ssl_.py in _ssl_wrap_socket_impl
urllib3/connection.py in connect
urllib3/connectionpool.py in _validate_conn
urllib3/poolmanager.py in urlopen
lightning_sdk/lightning_cloud/openapi/rest.py in request
```

## Expected behavior

Lightning SDK network operations should have a finite, configurable default connect/read timeout when callers do not provide `_request_timeout`.

Explicit per-request timeouts should continue to override that default.

At minimum, when no SDK timeout is configured, the REST client should avoid passing an explicit `timeout=None` so that urllib3 does not override the process socket default.

## Additional retry consideration

The timeout currently applies to one urllib3 attempt, not one logical SDK operation. urllib3 defaults to three retries, and higher-level Lightning clients may apply another retry loop.

With a 30-second timeout, a persistently silent endpoint can therefore take approximately:

4 urllib3 attempts * 7 SDK attempts * 30 seconds
+ 31.5 seconds of SDK backoff
≈ 871.5 seconds

It would be helpful for the SDK to define the timeout and retry policy together, or expose a logical-call deadline, to avoid multiplying two independent retry layers.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with lightning_cloud/openapi/rest.py at the cited timeout conversion and urllib3 call sites, then run the localhost reproducer without Lightning credentials. Done means omitted _request_timeout no longer permits an indefinite handshake while explicit per-request timeouts still override the default; the retry interaction may need clarification.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
api, networking
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
68/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.