Self-hosted Sentry: ingestion pipeline (relay ↔ web) silently fails after ~18–24h uptime, recurring, no clear root cause found after extensive investigation

### Environment

self-hosted (https://develop.sentry.dev/self-hosted/)

### What are you trying to accomplish?

Keep our self-hosted Sentry ingestion pipeline running reliably. We're trying to diagnose and report a recurring failure where event ingestion silently stops completely, roughly every 18–24 hours, requiring a full manual restart of relay, web, and all Kafka-touching consumers to recover — with zero human interaction in between failures (confirmed it fails again overnight, unattended).

### How are you getting stuck?

relay stops successfully fetching project config from web, even though every individual component tests healthy in isolation:

• relay logs ERROR relay_server::services::health_check: Health check probe 'auth' failed


• relay logs a continuous, steady-state backlog (~150–270 entries, cycling) of:

  WARN relay_server::services::projects::source::upstream: error fetching project state <key>: deadline exceeded errors=0 pending=N tags.did_error=false tags.was_pending=true

• web responds correctly and fast to both unauthenticated (401 in ~0.1–0.4s) and simple authenticated requests tested via curl from another container (nginx) hitting web:9000 directly on the same Docker bridge network


• Kafka accepts manual produces instantly via kafka-console-producer throughout the outage


• No offsets advance on ingest-events / ingest-transactions / ingest-attachments during the outage — relay never forwards anything downstream, despite still returning 200/event-IDs at the public ingestion edge (nginx → relay /api/N/envelope/ and /store/)

Trigger evidence found: in taskscheduler (long-running, Celery-beat-equivalent):

%6|...|FAIL|sentry.taskworker.taskworker#producer-1| [thrd:kafka:9092/1001]: kafka:9092/1001: Disconnected: connection closed by peer: receive 0 after POLLIN (after 66577654ms in state UP)

66577654ms ≈ 18.5 hours — a long-lived Kafka client connection was in UP state for ~18.5h before the remote peer (Kafka) closed it, discovered only on next use, no proactive reconnect. This periodicity matches the observed failure interval stack-wide.

What we've ruled out (across incidents on 07-17, 07-21, 07-22, 07-24):

Open question: no broker-side or host-conntrack evidence explains why Kafka closed this connection at ~18h. Either a gap in librdkafka's keepalive/reconnect handling for long-idle producer connections, or Docker Engine silently dropping long-idle bridge/NAT state without notifying either endpoint. Would help to know if relay/taskworker's Kafka producer clients set explicit socket.keepalive.enable or an idle-connection-max equivalent, and whether there's a known interaction with Docker bridge networking under long uptime.

### Where in the product are you?

Other

### Link

_No response_

### DSN

https://d16b71ab7918441b9de71b1273a6de55@sentry.officebooking.net/3

### Version

26.7.0.dev0

Begin with the taskscheduler Kafka producer disconnect log and compare relay's project-config health-check and backlog behavior across an 18–24-hour run. Inspect the relay/taskworker producer connection settings and Docker bridge behavior; done means reproducing the failure and establishing a verified reconnect, keepalive, or networking fix with ingestion offsets advancing after long uptime.

Written by the indexing model from the issue text.