getsentry / getsentry/self-hosted
Relay stopped processing new metrics and events
- Dominant language
- Shell
- Stars
- 9.6k
- Forks
- 2k
- Avg merge
- 1d 3h
- Merged PRs (30d)
- 12
Description
### Self-Hosted Version
26.3.1
### CPU Architecture
x86_64
### Docker Version
29.2.1
### Docker Compose Version
5.1.0
### Machine Specification
- [x] My system meets the minimum system requirements of Sentry
### Installation Type
Upgrade
### Steps to Reproduce
1. Start the whole stack using docker compose
2. Wait
### Expected Result
Sentry should keep on keeping on.
### Actual Result
Sentry stops processing new metrics and events.
Looking at logs, we get tons of entries similar to the ones below in the `relay` container, and the start of those entries matches the end of processing.
```
relay-1 | 2026-03-24T12:44:30.887984Z ERROR relay_kafka::producer::utils: failed to produce message to Kafka (delivery callback) error=Message production error: MessageTimedOut (Local: Message timed out) error.sources=[MessageTimedOut (Local: Message timed out)] payload_len=45808 tags.topic="ingest-events"
```
These errors are produced for all kinds of topics.
Restarting the relay container makes things move again, but in flight data is lost :(
### Event ID
_No response_
Contributor guide
Research direction
Start the stack with Docker Compose and inspect the relay container logs around relay_kafka::producer::utils and the MessageTimedOut delivery errors. Reproduce the stoppage across the affected Kafka topics and trace when processing ends. Done means relay continues processing metrics and events without recurring production timeouts or losing in-flight data after an upgrade.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- docker, docker-compose, kafka
- Domain
- backend, distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100