getsentry / getsentry/sentry

Taskworker: high fetch_task.not_found and consumer lag when broker returns empty GetTask (no rebalance)

Open
#114,018 3 comments 0 reactions 0 assignees View on GitHub
Bug Waiting for: Product Owner
Dominant language
Python
Stars
44.8k
Forks
4.9k
Avg merge
21h 10m
Merged PRs (30d)
635

Description

### Self-Hosted Version

26.2.1

### CPU Architecture

x86_64

### Docker Version

unrelated

### Docker Compose Version

unrelated

### Machine Specification

- [x] My system meets the minimum system requirements of Sentry

### Installation Type

Upgrade from 25.12.1

### Steps to Reproduce

### How can we see what you're seeing? Specific is terrific.
1. Deploy Sentry with taskbroker + taskworker (Kafka topic taskworker, 16 partitions), multiple taskbroker replicas and taskworker workers.
2. Create normal event/task load so the taskworker topic has steady traffic.
3. Scrape metrics from the StatsD/Prometheus exporter (e.g. curl -s localhost:9102/metrics) and watch Kafka consumer lag for the taskworker topic.
4. Observe: taskbroker_consumer_inflight_activation_writer_backpressure{reason="pending_limit"} grows, sentry_taskworker_worker_fetch_task_not_found is high (tens of thousands), consumer lag on taskworker increases.
5. In code, confirm that when GetTask returns success with no task field, the client in sentry/taskworker/client/client.py returns None and does not set _num_tasks_before_rebalance = 0.

### Expected Result

1. When a broker has no pending task, the worker should quickly switch to another broker (rebalance), so it does not keep polling the same “empty” broker.
2. fetch_task.not_found should stay relatively low; pending tasks should be drained across brokers and consumer lag should not grow without bound when capacity is sufficient.

### Actual Result

### Metrics:
```
taskbroker_consumer_inflight_activation_writer_backpressure{reason="pending_limit"} ≈ 158k and increasing
taskbroker_upkeep_current_pending_tasks ≈ 846
taskbroker_upkeep_current_processing_tasks = 0
taskbroker_upkeep_pending_activation_max_lag_sec ≈ 277315 (~77 hours)
sentry_taskworker_worker_fetch_task_not_found ≈ 69k

```
### Kafka consumer lag on topic taskworker — growing

Observed behavior:
Workers stick to one broker. When that broker returns GetTask success with no task (empty response), the client does not rebalance, so the worker keeps polling the same broker and increments fetch_task.not_found every time. Rebalance only happens on gRPC NOT_FOUND or after max_tasks_before_rebalance completed tasks. As a result, many workers hit “empty” brokers, pending is not drained evenly, taskbroker hits pending_limit, stops reading from Kafka, and lag grows.

### Relevant code:

sentry/taskworker/client/client.py — in get_task(), when not response.HasField("task") the code returns None and does not set self._num_tasks_before_rebalance = 0.

### Event ID

_No response_

Contributor guide

Open the contributing guide

Research direction

Start in sentry/taskworker/client/client.py, read get_task(), and trace how an empty GetTask response affects rebalance state. Reproduce or inspect the reported metrics and worker behavior; done means empty broker responses no longer keep workers polling one broker, fetch_task.not_found remains low, and consumer lag does not grow with sufficient capacity.

Written by the indexing model from the issue text.

Assessment

Tech stack
kafka, python
Domain
backend, distributed-systems
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Clearly specified
Newbie friendliness
58/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.