getsentry / getsentry/self-hosted

Multiple Sentry consumers repeatedly become unhealthy due to Kafka coordinator/session timeouts

Open
#4,485 4 comments 0 reactions 0 assignees View on GitHub
Waiting for: Product Owner
Dominant language
Shell
Stars
9.6k
Forks
2k
Avg merge
1d 5h
Merged PRs (30d)
15

Description

### Self-Hosted Version

26.8

### CPU Architecture

x86_64

### Docker Version

29.7.2

### Docker Compose Version

v5.5.0

### Machine Specification

- [x] My system meets the minimum system requirements of Sentry

### Installation Type

Upgrade from 26.7.2

### Steps to Reproduce

1. Deploy a self-hosted Sentry installation using the standard Docker Compose configuration.
2. Start all Sentry services and verify that the consumers initially become healthy.
3. Leave the Sentry installation running under normal workload.
4. Allow the Sentry consumers to process events, performance metrics, buffered segments, monitor tasks, and other Kafka topics.
5. After some time, multiple consumers begin losing communication with the Kafka consumer group coordinator.
6. The issue can be observed in the logs with messages such as:

```text
Consumer group session timed out (in join-state steady) after 45000 ms
without a successful response from the group coordinator
```
7. The consumers subsequently report errors such as:

```text
KafkaError{code=NOT_COORDINATOR}
KafkaError{code=_WAIT_COORD}
KafkaError{code=COORDINATOR_LOAD_IN_PROGRESS}
```
8. The affected containers eventually become `unhealthy`.
9. Restarting the Sentry stack temporarily restores the consumers to a healthy state.
10. The problem recurs several times per day, requiring approximately 4 Sentry restarts per day.

Affected containers include:

```text
sentry-snuba-subscription-consumer-events-1
sentry-process-segments-1
sentry-post-process-forwarder-errors-1
sentry-monitors-clock-tasks-1
sentry-ingest-monitors-1
sentry-generic-metrics-consumer-1
sentry-events-consumer-1
```

The issue also affects Snuba's Kafka consumer for:

```text
snuba-commit-log
```

### Expected Result

Sentry consumers should remain connected to their Kafka consumer groups and continue processing messages normally during continuous operation.

The consumer containers should remain `healthy` and should not require manual restarts.

Temporary Kafka coordinator changes or consumer group rebalances should recover automatically without causing the containers to become `unhealthy`.

### Actual Result

Multiple Sentry consumers periodically lose communication with the Kafka group coordinator and experience approximately 45-second consumer session timeouts.

The logs contain messages such as:

```text
Consumer group session timed out (in join-state steady) after 45000 ms
without a successful response from the group coordinator
```

followed by:

```text
Broker: Not coordinator
```

```text
Local: Waiting for coordinator
```

and:

```text
Broker: Coordinator load in progress
```

The consumers revoke their Kafka partitions and attempt to rejoin the consumer group.

For example:

```text
Partitions to revoke:
[Partition(topic=Topic(name='events'), index=0)]
```

and:

```text
Commit failed:
KafkaError{code=_WAIT_COORD,val=-180,str="Local: Waiting for coordinator"}
```

Snuba exhibits the same behavior on `snuba-commit-log`, suggesting that the problem is not isolated to a single Sentry consumer.

Additionally, `sentry-process-segments-1` frequently reports processing delays such as:

```text
Batch process is taking longer than 5 seconds.
```

and:

```text
Waited on the process pool longer than 20 seconds.
Waiting for 1 results.
```

After these Kafka coordinator/session problems, multiple containers become `unhealthy`.

Restarting the Sentry stack temporarily resolves the problem, but the same behavior returns several times per day, requiring approximately **4 manual restarts per day**.

[_sentry-process-segments-1_logs.txt](https://github.com/user-attachments/files/31309407/_sentry-process-segments-1_logs.txt)
[_sentry-post-process-forwarder-errors-1_logs.txt](https://github.com/user-attachments/files/31309404/_sentry-post-process-forwarder-errors-1_logs.txt)
[_sentry-monitors-clock-tasks-1_logs.txt](https://github.com/user-attachments/files/31309402/_sentry-monitors-clock-tasks-1_logs.txt)
[_sentry-ingest-monitors-1_logs.txt](https://github.com/user-attachments/files/31309406/_sentry-ingest-monitors-1_logs.txt)
[_sentry-generic-metrics-consumer-1_logs.txt](https://github.com/user-attachments/files/31309405/_sentry-generic-metrics-consumer-1_logs.txt)
[_sentry-events-consumer-1_logs.txt](https://github.com/user-attachments/files/31309403/_sentry-events-consumer-1_logs.txt)
[_sentry-snuba-subscription-consumer-events-1_logs.txt](https://github.com/user-attachments/files/31309401/_sentry-snuba-subscription-consumer-events-1_logs.txt)

### Event ID

_No response_

Contributor guide

Open the contributing guide

Research direction

Start with the attached logs and the standard Docker Compose configuration, focusing on the Kafka coordinator timeouts, consumer rebalances, and unhealthy-container transitions across Sentry and Snuba consumers. Compare the affected consumer services and identify a reproducible configuration or recovery change; done means consumers remain healthy and process messages continuously without manual restarts.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker, docker-compose, kafka
Domain
devops, distributed-systems, infrastructure
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.