getsentry / getsentry/self-hosted

ClickhousePool: stale idle connections cause 30s query hangs; tcp_keepalive hard-coded off and not configurable

Open
#4,440 0 comments 0 reactions 0 assignees View on GitHub
Waiting for: Product Owner
Dominant language
Shell
Stars
9.6k
Forks
2k
Avg merge
1d 5h
Merged PRs (30d)
15

Description

### Self-Hosted Version

26.4.2

### CPU Architecture

x86_64

### Docker Version

29.6.1

### Docker Compose Version

v5.3.0

### Machine Specification

- [x] My system meets the minimum system requirements of Sentry

### Installation Type

Upgrade of an existing install

### Steps to Reproduce

1. Have a handful of Uptime monitors configured.
2. Open the Uptime monitors page after snuba-api has been running a while with
low query traffic.
3. The page fires one `OrganizationUptimeStatsEndpoint` /
`OrganizationUptimeSummaryEndpoint` EAP RPC per monitor, concurrently.

Root cause (traced in code): `snuba/clickhouse/native.py::ClickhousePool` keeps a
LIFO pool of persistent clickhouse-driver connections. Under low traffic the
connections at the bottom of the LIFO pool sit idle long enough to be silently
dropped by the network (no FIN/RST) and remain in the pool as half-open sockets.
The concurrent uptime burst checks out several connections at once, reaching
those stale ones. `_create_conn()` builds `Client(...)` with `tcp_keepalive`
defaulting to `False` and there is no pre-checkout liveness check, so a query on
a dead connection blocks until `send_receive_timeout` (hard-coded 35s) instead of
failing fast. The web read timeout to snuba-api is 30s, so it aborts first. One
poisoned connection degrades the pool for a long time.

Same error as #2151 (closed "not planned", "reloading fixes it"); this adds the
missing diagnosis.
Expected Result

The Uptime page loads. A pool connection that was silently dropped while idle
should not block a query for ~30s; it should be detected (TCP keepalive or a
cheap pre-checkout ping) and replaced.
Actual Result

Every uptime-stats / uptime-summary call returns HTTP 400 after ~30s; the page
spins forever. Over one 2h window: 51x HTTP 400 (all ~30s) vs 2x HTTP 200.

Web:
SnubaRPCError: HTTPConnectionPool(host='snuba-api', port=1218): Read timed out. (read timeout=30)
sentry.uptime.endpoints.organization_uptime_stats: Error making EAP RPC request for uptime check stats

snuba-api (correlated):
Error on clickhouse:9000 ping: Unexpected EOF while reading bytes
Connection was closed, reconnecting.

Not a data-volume problem (distinct from #3952): eap_items_1_local is 2.9M rows /
564 MiB and the same aggregation runs in ~0.1s directly on ClickHouse. snuba-api
is idle, no OOM, ClickHouse healthy. Reproducing the exact EAP timeseries_rpc
sequentially from a `sentry django shell` succeeds in ~0.1s; the hang only
appears under the concurrent burst that reaches stale pool connections.

Requested fix (any): (1) plumb/enable `tcp_keepalive` on the driver Client from a
setting - without SO_KEEPALIVE, OS-level net.ipv4.tcp_keepalive_* has no effect,
so this can't be worked around from config; (2) validate/evict idle connections
before checkout; (3) at minimum expose `send_receive_timeout` so it can be set
below 30s and let the existing execute_robust retry recover.

### Expected Result

The Uptime page loads. A pool connection that was silently dropped while idle
should not block a query for ~30s; it should be detected (TCP keepalive or a
cheap pre-checkout ping) and replaced.

### Actual Result

Every uptime-stats / uptime-summary call returns HTTP 400 after ~30s; the page
spins forever. Over one 2h window: 51x HTTP 400 (all ~30s) vs 2x HTTP 200.

Web:
SnubaRPCError: HTTPConnectionPool(host='snuba-api', port=1218): Read timed out. (read timeout=30)
sentry.uptime.endpoints.organization_uptime_stats: Error making EAP RPC request for uptime check stats

snuba-api (correlated):
Error on clickhouse:9000 ping: Unexpected EOF while reading bytes
Connection was closed, reconnecting.

Not a data-volume problem (distinct from #3952): eap_items_1_local is 2.9M rows /
564 MiB and the same aggregation runs in ~0.1s directly on ClickHouse. snuba-api
is idle, no OOM, ClickHouse healthy. Reproducing the exact EAP timeseries_rpc
sequentially from a `sentry django shell` succeeds in ~0.1s; the hang only
appears under the concurrent burst that reaches stale pool connections.

Requested fix (any): (1) plumb/enable `tcp_keepalive` on the driver Client from a
setting - without SO_KEEPALIVE, OS-level net.ipv4.tcp_keepalive_* has no effect,
so this can't be worked around from config; (2) validate/evict idle connections
before checkout; (3) at minimum expose `send_receive_timeout` so it can be set
below 30s and let the existing execute_robust retry recover.

### Event ID

_No response_

Contributor guide

Open the contributing guide

Research direction

Start in snuba/clickhouse/native.py at ClickhousePool and _create_conn(), then trace checkout behavior and the existing execute_robust retry path. Compare the requested tcp_keepalive, pre-checkout validation, and send_receive_timeout options against the observed 30-second timeout. Done means stale idle connections no longer make the concurrent uptime requests hang and the pool recovers them without exceeding the web read timeout.

Written by the indexing model from the issue text.

Assessment

Tech stack
clickhouse, python
Domain
backend, databases, performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.