apache / apache/pulsar

[Bug] REST API publisher by-passes producer rate limiting / memory limits and lacks back pressure

Open
#24,229 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
15.3k
Forks
3.8k
Avg merge
1d 14h
Merged PRs (30d)
160

Description

> Another problem is flow control / back pressure. The REST publisher by-passes all ordinary ways that Pulsar broker handles back pressure for clients and that would result in OOMEs too when clients push more messages than the broker can keep up with. For example, the `maxMessagePublishBufferSizeInMB` gets ignored completely.
>

_Originally posted by @lhotari in [#24227](https://github.com/apache/pulsar/issues/24227#issuecomment-2838192701)_

feature was introduced by "PIP-64: [REST] Rest API Produce message" #8125

Contributor guide

Open the contributing guide

Research direction

Start by tracing the REST publisher introduced by PIP-64: REST API Produce message. Investigate how maxMessagePublishBufferSizeInMB and the broker's ordinary back-pressure mechanisms apply to this path. Done means REST publishing respects rate and memory limits and provides flow control without allowing unbounded buffering or OOMEs.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
api, backend, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.