Taskworker: high fetch_task.not_found and consumer lag when broker returns empty GetTask (no rebalance)
- Dominant language
- Python
- Stars
- 44.8k
- Forks
- 4.9k
- Avg merge
- 21h 10m
- Merged PRs (30d)
- 635
Description
### Self-Hosted Version
26.2.1
### CPU Architecture
x86_64
### Docker Version
unrelated
### Docker Compose Version
unrelated
### Machine Specification
- [x] My system meets the minimum system requirements of Sentry
### Installation Type
Upgrade from 25.12.1
### Steps to Reproduce
### How can we see what you're seeing? Specific is terrific.
1. Deploy Sentry with taskbroker + taskworker (Kafka topic taskworker, 16 partitions), multiple taskbroker replicas and taskworker workers.
2. Create normal event/task load so the taskworker topic has steady traffic.
3. Scrape metrics from the StatsD/Prometheus exporter (e.g. curl -s localhost:9102/metrics) and watch Kafka consumer lag for the taskworker topic.
4. Observe: taskbroker_consumer_inflight_activation_writer_backpressure{reason="pending_limit"} grows, sentry_taskworker_worker_fetch_task_not_found is high (tens of thousands), consumer lag on taskworker increases.
5. In code, confirm that when GetTask returns success with no task field, the client in sentry/taskworker/client/client.py returns None and does not set _num_tasks_before_rebalance = 0.
### Expected Result
1. When a broker has no pending task, the worker should quickly switch to another broker (rebalance), so it does not keep polling the same “empty” broker.
2. fetch_task.not_found should stay relatively low; pending tasks should be drained across brokers and consumer lag should not grow without bound when capacity is sufficient.
### Actual Result
### Metrics:
```
taskbroker_consumer_inflight_activation_writer_backpressure{reason="pending_limit"} ≈ 158k and increasing
taskbroker_upkeep_current_pending_tasks ≈ 846
taskbroker_upkeep_current_processing_tasks = 0
taskbroker_upkeep_pending_activation_max_lag_sec ≈ 277315 (~77 hours)
sentry_taskworker_worker_fetch_task_not_found ≈ 69k
```
### Kafka consumer lag on topic taskworker — growing
Observed behavior:
Workers stick to one broker. When that broker returns GetTask success with no task (empty response), the client does not rebalance, so the worker keeps polling the same broker and increments fetch_task.not_found every time. Rebalance only happens on gRPC NOT_FOUND or after max_tasks_before_rebalance completed tasks. As a result, many workers hit “empty” brokers, pending is not drained evenly, taskbroker hits pending_limit, stops reading from Kafka, and lag grows.
### Relevant code:
sentry/taskworker/client/client.py — in get_task(), when not response.HasField("task") the code returns None and does not set self._num_tasks_before_rebalance = 0.
### Event ID
_No response_
Contributor guide
Research direction
Start in sentry/taskworker/client/client.py, read get_task(), and trace how an empty GetTask response affects rebalance state. Reproduce or inspect the reported metrics and worker behavior; done means empty broker responses no longer keep workers polling one broker, fetch_task.not_found remains low, and consumer lag does not grow with sufficient capacity.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- kafka, python
- Domain
- backend, distributed-systems
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Clearly specified
- Newbie friendliness
- 58/100