apache / apache/pulsar-connectors

[Enhancement] Add Backpressure Handling in Kafka Connector Adaptor to Prevent OOM Under Sink Stall Conditions

Open
#15 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
26
Forks
25
Avg merge
17h 7m
Merged PRs (30d)
1

Description

### Search before reporting

- [x] I searched in the [issues](https://github.com/apache/pulsar-connectors/issues) and found nothing similar.

### Read release policy

- [x] I understand that [unsupported versions](https://pulsar.apache.org/contribute/release-policy/#supported-versions) don't get bug fixes. I will attempt to reproduce the issue on a supported version of Pulsar client and Pulsar broker.

### User environment

master

### Issue Description

The `KafkaConnectAdapter` currently lacks an effective backpressure mechanism to regulate message ingestion and buffering. As a result, it continues consuming records and appending them to the `pendingFlushQueue` even when downstream processing is saturated or stalled.

At present, `batchSize` is only used as a flush threshold and does not impose any limit on writing records. This leads to accumulation of records in memory, which can result in OutOfMemoryError (OOM) under sustained load or when the sink is slow/unavailable.

### Error messages

```text

```

### Reproducing the issue

Run any sink using KafkaConnectAdapter (e.g., BigQuery sink) and simulate a slow or stalled downstream. Observe unbounded growth of `pendingFlushQueue` leading to OOM.

### Additional information

_No response_

### Are you willing to submit a PR?

- [x] I'm willing to submit a PR!

Contributor guide

No contributing guide indexed for this repository

Research direction

Start at KafkaConnectAdapter and trace how records enter and leave pendingFlushQueue. Run a sink such as BigQuery with a slow or stalled downstream to observe queue growth. Done means ingestion is regulated under sink saturation and the pending queue no longer grows without bound toward an OOM.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, kafka
Domain
stream-processing
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.