apache / apache/pulsar

[Bug] Pulsar shutdowns Sink connector on exception

Open
#19,894 18 comments 0 reactions 0 assignees View on GitHub
Stale type/bug
Dominant language
Java
Stars
15.3k
Forks
3.8k
Avg merge
1d 14h
Merged PRs (30d)
160

Description

### Search before asking

- [X] I searched in the [issues](https://github.com/apache/pulsar/issues) and found nothing similar.

### Version

* Pulsar 2.7.3 onwards

### Minimal reproduce step

* Change any sink implementation for test and call `record.fail()` from the write method. The processing guarantee for the sink should be set to `EFFECTIVELY_ONCE` and auto ack should be `false`.

### What did you expect to see?

For the sinks having `EFFECTIVE_ONCE` guarantee, prior to pulsar v2.7.3 , if there was any exception in record processing, the `CompletableFuture` [callback](https://github.com/apache/pulsar/blob/v2.7.2/pulsar-functions/instance/src/main/java/org/apache/pulsar/functions/instance/JavaInstanceRunnable.java#L345) was swallowing the [RuntimeException](https://github.com/apache/pulsar/blob/v2.7.2/pulsar-functions/instance/src/main/java/org/apache/pulsar/functions/instance/JavaInstanceRunnable.java#L376). From v2.7.3, it is [not processed in the callback](https://github.com/apache/pulsar/blob/v2.7.3/pulsar-functions/instance/src/main/java/org/apache/pulsar/functions/instance/JavaInstanceRunnable.java#L349) hence [caught](https://github.com/apache/pulsar/blob/v2.7.3/pulsar-functions/instance/src/main/java/org/apache/pulsar/functions/instance/JavaInstanceRunnable.java#L277) by the `run` method of `JavaInstanceRunnable` which shutdowns the sink. Is this expected behavior? In this case, on failure, no further processing will happen as the sink is shutdown. Shouldn't this exception be handled and the record be retried and allow other future message processing as well? At least, there should be a config option that would decide to shutdown the sink or keep it running.

### What did you see instead?

Pulsar shutdowns the sink service hence that particular record and any further records are not processed by the sink.

### Anything else?

_No response_

### Are you willing to submit a PR?

- [X] I'm willing to submit a PR!

Contributor guide

Open the contributing guide

Research direction

The linked JavaInstanceRunnable locations in v2.7.2 and v2.7.3 are the entry points; compare callback handling with the run method. Reproduce using a sink that calls record.fail() with EFFECTIVELY_ONCE and auto ack false, then use the expected retry-versus-shutdown behavior to define the regression test and completion criteria.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
distributed-systems, stream-processing
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.