apache / apache/pulsar

[BUG] Custom Java Sink Connector: Logging statement in 'close' method not executed on deletion or stop

Open
#23,063 6 comments 0 reactions 0 assignees View on GitHub
type/bug
Dominant language
Java
Stars
15.3k
Forks
3.8k
Avg merge
1d 14h
Merged PRs (30d)
160

Description

When adding a logging statement inside the "close" method of a custom sink connector, uploading it to a standalone cluster, and then deleting or stopping the connector, the logging statement is not executed as expected.

### Version

OS: Arch Linux x86_64
Kernel: 6.6.40-1-lts
Java:
- openjdk 17.0.12 2024-07-16
- OpenJDK Runtime Environment (build 17.0.12+7)
- OpenJDK 64-Bit Server VM (build 17.0.12+7, mixed mode, sharing)

Apache Pulsar: 3.3.0 (using apachepulsar/pulsar:3.3.0 Docker image)

Java Client Libraries:

- org.apache.pulsar:pulsar-client-original:3.3.0
- org.apache.pulsar:pulsar-client-admin-original:3.3.0

### Minimal reproduce step

Steps to reproduce:
1. Set up a standalone Apache Pulsar cluster using Docker Compose. Docker Compose configuration:

```yaml
services:
pulsar:
container_name: pulsar
image: apachepulsar/pulsar:3.3.0
ports:
- "6650:6650"
- "8080:8080"
- "8081:8081"
environment:
- PULSAR_STANDALONE_USE_ZOOKEEPER=1
command: ["bin/pulsar", "standalone"]
```

2. Implement a custom sink connector with logging statements in "open" and "close" methods. Sink Connector:
```java
public class SimpleSink implements Sink {
private Logger LOG = null;
@Override
public void open(Map config, SinkContext sinkContext) {
LOG = sinkContext.getLogger();
LOG.info("Hello world");
}
@Override
public void write(Record record) {
String value = record.getValue();
LOG.info("Recorded {}.", value);
record.ack();
}
@Override
public void close() {
LOG.info("Bye world");
}
}
```
3. Upload the sink connector to the Pulsar cluster. Sink Connector Configuration:
```yaml
tenant: "simple-tenant"
namespace: "simple-ns"
name: "DataSink"
className: "dev.nurramo.sink.SimpleSink"
inputs: ["persistent://simple-tenant/simple-ns/output"]
sourceSubscriptionName: "sub-1"

```
4. Stop or delete the connector
5. Check logs for the expected logging statements

### What did you expect to see?

The logging statements in both the "open" and "close" methods should be executed and visible in the logs when the connector is started and then stopped or deleted.

### What did you see instead?

Only the logging statement from the "open" method is visible in the logs. The "close" method's logging statement is not executed or logged when the connector is stopped or deleted.

Here is how I confirmed that. :
```bash
/pulsar/logs/functions/simple-tenant/simple-ns/DataSink $ cat *.log | grep world
2024-07-22T21:42:01,615+0000 [simple-tenant/simple-ns/DataSink-0] INFO function-DataSink - Hello world
```
"Bye world" was not listed anywhere

### Anything else?

I haven't encountered any issues running the sink via the Localrunner. Both "Hello world" and "Bye world" pop up in the log output.

### Are you willing to submit a PR?

- [ ] I'm willing to submit a PR!

### Search before asking

- [X] I searched in the [issues](https://github.com/apache/pulsar/issues) and found nothing similar.

### Read release policy

- [X] I understand that unsupported versions don't get bug fixes. I will attempt to reproduce the issue on a supported version of Pulsar client and Pulsar broker.

Contributor guide

Open the contributing guide

Research direction

Reproduce the issue with the provided Docker Compose standalone setup and the SimpleSink implementation, then compare connector behavior with the Localrunner, where both messages appear. Inspect the connector stop and deletion path, using the function logs to verify lifecycle output. Done means the close method's "Bye world" message appears when the sink is stopped or deleted.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker-compose, java
Domain
backend, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.