apache / apache/rocketmq-clients
[Enhancement][Python] Make PushConsumer message reception non-blocking to support large numbers of topics
- Dominant language
- Java
- Stars
- 505
- Forks
- 313
- Avg merge
- 11h 28m
- Merged PRs (30d)
- 6
Description
### Before Creating the Enhancement Request
- [x] I have confirmed that this should be classified as an enhancement rather than a bug/feature.
### Programming Language of the Client
Python
### Summary
The Python PushConsumer submits a blocking receive call to a ThreadPoolExecutor, so every in-flight long-polling request occupies one worker until the server-stream completes.
Concurrent reception is therefore bounded by the pool size rather than by the number of assigned queues, and with many subscribed topics most queues wait many long-polling windows before their first receive — appearing effectively starved in production. The log repeats:
> 2026-08-26 11:44:16,321 [WARNING] process queue is idle, idle_duration: 1022, max_idle_duration: 69, after_cache_full_duration: 1787715857, mq: broker-0.Topic100.-1
2026-08-26 11:44:16,332 [WARNING] process queue is idle, idle_duration: 1022, max_idle_duration: 69, after_cache_full_duration: 1787715857, mq: broker-0.Topic100.-1
2026-08-26 11:44:16,344 [WARNING] process queue is idle, idle_duration: 1022, max_idle_duration: 69, after_cache_full_duration: 1787715857, mq: broker-0.Topic100.-1
2026-08-26 11:44:16,355 [WARNING] process queue is idle, idle_duration: 1022, max_idle_duration: 69, after_cache_full_duration: 1787715857, mq: broker-0.Topic100.-1
2026-08-26 11:44:16,366 [WARNING] process queue is idle, idle_duration: 1022, max_idle_duration: 69, after_cache_full_duration: 1787715857, mq: broker-0.Topic100.-1
2026-08-26 11:44:16,377 [WARNING] process queue is idle, idle_duration: 1022, max_idle_duration: 69, after_cache_full_duration: 1787715857, mq: broker-0.Topic100.-1
### Motivation
When a single PushConsumer subscribes to 100 topics, messages on most of those topics are effectively never consumed.
The consumer stays up and healthy and a few topics work normally, but the rest produce no deliveries for as long as the process runs — only the process queue is idle warning above, repeating indefinitely.
### Describe the Solution You'd Like
**Make reception non-blocking**. Replace the unbounded-default reception pool with a fully callback-driven receive path that does not occupy one executor worker per in-flight receive.
**Fix the related defects that keep starved queues from recovering**. These sit on the same paths and are best fixed together:
- The expired-queue cleanup has an inverted condition.
- The assignment handler starts reception for all existing queues rather than only the newly created ones.
- The non-FIFO consumption path misses a return after discarding a corrupted message.
- The FIFO discard path also lacks a return after its recursive retry.
### Describe Alternatives You've Considered
/
### Additional Context
_No response_
Contributor guide
No contributing guide indexed for this repository
Research direction
Start at the Python PushConsumer receive path and its ThreadPoolExecutor usage, then trace queue assignment and the FIFO and non-FIFO discard paths. Done means reception no longer starves queues as subscribed topics increase and the expired-queue, assignment, and discard-path defects are corrected without breaking message delivery.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- distributed-systems
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100