getsentry / getsentry/self-hosted

Relay stopped processing new metrics and events

Open
#4,240 35 comments 7 reactions 0 assignees View on GitHub
Waiting for: Product Owner
Dominant language
Shell
Stars
9.6k
Forks
2k
Avg merge
1d 3h
Merged PRs (30d)
12

Description

### Self-Hosted Version

26.3.1

### CPU Architecture

x86_64

### Docker Version

29.2.1

### Docker Compose Version

5.1.0

### Machine Specification

- [x] My system meets the minimum system requirements of Sentry

### Installation Type

Upgrade

### Steps to Reproduce

1. Start the whole stack using docker compose
2. Wait

### Expected Result

Sentry should keep on keeping on.

### Actual Result

Sentry stops processing new metrics and events.

Looking at logs, we get tons of entries similar to the ones below in the `relay` container, and the start of those entries matches the end of processing.

```
relay-1 | 2026-03-24T12:44:30.887984Z ERROR relay_kafka::producer::utils: failed to produce message to Kafka (delivery callback) error=Message production error: MessageTimedOut (Local: Message timed out) error.sources=[MessageTimedOut (Local: Message timed out)] payload_len=45808 tags.topic="ingest-events"
```

These errors are produced for all kinds of topics.

Restarting the relay container makes things move again, but in flight data is lost :(

### Event ID

_No response_

Contributor guide

Open the contributing guide

Research direction

Start the stack with Docker Compose and inspect the relay container logs around relay_kafka::producer::utils and the MessageTimedOut delivery errors. Reproduce the stoppage across the affected Kafka topics and trace when processing ends. Done means relay continues processing metrics and events without recurring production timeouts or losing in-flight data after an upgrade.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker, docker-compose, kafka
Domain
backend, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.