[Bug] REST API publisher by-passes producer rate limiting / memory limits and lacks back pressure
- Dominant language
- Java
- Stars
- 15.3k
- Forks
- 3.8k
- Avg merge
- 1d 14h
- Merged PRs (30d)
- 160
Description
> Another problem is flow control / back pressure. The REST publisher by-passes all ordinary ways that Pulsar broker handles back pressure for clients and that would result in OOMEs too when clients push more messages than the broker can keep up with. For example, the `maxMessagePublishBufferSizeInMB` gets ignored completely.
>
_Originally posted by @lhotari in [#24227](https://github.com/apache/pulsar/issues/24227#issuecomment-2838192701)_
feature was introduced by "PIP-64: [REST] Rest API Produce message" #8125
Contributor guide
Research direction
Start by tracing the REST publisher introduced by PIP-64: REST API Produce message. Investigate how maxMessagePublishBufferSizeInMB and the broker's ordinary back-pressure mechanisms apply to this path. Done means REST publishing respects rate and memory limits and provides flow control without allowing unbounded buffering or OOMEs.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- api, backend, distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100