apache / apache/pulsar

Flaky-test: PulsarFunctionTlsTest "Leader not yet ready"

Open
#23,385 2 comments 0 reactions 0 assignees View on GitHub
area/test type/flaky-tests
Dominant language
Java
Stars
15.3k
Forks
3.8k
Avg merge
1d 14h
Merged PRs (30d)
160

Description

### Search before asking

- [X] I searched in the [issues](https://github.com/apache/pulsar/issues) and found nothing similar.

### Example failure

https://github.com/apache/pulsar/actions/runs/11120172357/job/30897034288?pr=23327#step:11:1154

### Exception stacktrace

```
org.apache.pulsar.client.admin.PulsarAdminException: javax.ws.rs.ServiceUnavailableException: HTTP 503 {"reason":"Leader not yet ready. Please retry again"}
at org.apache.pulsar.client.admin.PulsarAdminException.wrap(PulsarAdminException.java:252)
at org.apache.pulsar.client.admin.internal.BaseResource.sync(BaseResource.java:352)
at org.apache.pulsar.client.admin.internal.FunctionsImpl.createFunctionWithUrl(FunctionsImpl.java:199)
at org.apache.pulsar.functions.worker.PulsarFunctionTlsTest.testFunctionsCreation(PulsarFunctionTlsTest.java:274)
at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
at java.base/java.lang.reflect.Method.invoke(Method.java:580)
at org.testng.internal.invokers.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:139)
at org.testng.internal.invokers.InvokeMethodRunnable.runOne(InvokeMethodRunnable.java:47)
at org.testng.internal.invokers.InvokeMethodRunnable.call(InvokeMethodRunnable.java:76)
at org.testng.internal.invokers.InvokeMethodRunnable.call(InvokeMethodRunnable.java:11)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
at java.base/java.lang.Thread.run(Thread.java:1583)
```

```
2024-10-01T07:10:59,801 - INFO - [pulsar-external-listener-3425-1:WorkerUtils] - Encountered exception while at creating exclusive producer to topic persistent://public/functions/assignment
java.util.concurrent.ExecutionException: org.apache.pulsar.client.api.PulsarClientException$ProducerFencedException: {"errorMsg":"org.apache.pulsar.broker.service.BrokerServiceException$ProducerFencedException: Topic has an existing exclusive producer: c-my-cluster-fw-localhost-46303-scheduler-manager","reqId":2002931073644165119, "remote":"localhost/127.0.0.1:43319", "local":"/127.0.0.1:34620"}
at java.base/java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:396) ~[?:?]
at java.base/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2096) ~[?:?]
at org.apache.pulsar.functions.worker.WorkerUtils.createExclusiveProducerWithRetry(WorkerUtils.java:409) ~[pulsar-functions-worker-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
at org.apache.pulsar.functions.worker.SchedulerManager.acquireExclusiveWrite(SchedulerManager.java:171) ~[pulsar-functions-worker-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
at org.apache.pulsar.functions.worker.LeaderService.becameActive(LeaderService.java:112) ~[pulsar-functions-worker-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
at org.apache.pulsar.client.impl.ConsumerImpl.lambda$activeConsumerChanged$31(ConsumerImpl.java:1201) ~[pulsar-client-original-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?]
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?]
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[netty-common-4.1.113.Final.jar:4.1.113.Final]
at java.base/java.lang.Thread.run(Thread.java:1583) [?:?]
Caused by: org.apache.pulsar.client.api.PulsarClientException$ProducerFencedException: {"errorMsg":"org.apache.pulsar.broker.service.BrokerServiceException$ProducerFencedException: Topic has an existing exclusive producer: c-my-cluster-fw-localhost-46303-scheduler-manager","reqId":2002931073644165119, "remote":"localhost/127.0.0.1:43319", "local":"/127.0.0.1:34620"}
```

full log output: https://gist.github.com/lhotari/bdd307df2e28331e033d0fe4f30ca5f0

### Are you willing to submit a PR?

- [ ] I'm willing to submit a PR!

Contributor guide

Open the contributing guide

Research direction

Start with PulsarFunctionTlsTest.testFunctionsCreation at line 274 and trace the failure through FunctionsImpl.createFunctionWithUrl. Then inspect WorkerUtils.createExclusiveProducerWithRetry, SchedulerManager.acquireExclusiveWrite, and LeaderService.becameActive for the leader-readiness and producer-fencing interaction. Run the affected PulsarFunctionTlsTest and confirm it no longer fails with HTTP 503 while the worker is becoming ready.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
distributed-systems, testing-qa
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.