apache / apache/pulsar

Attempting to delete persistent topics shortly after they are created yields intermittent "BKException$BKUnexpectedConditionException: Unexpected condition"

Open
#13,172 2 comments 0 reactions 0 assignees View on GitHub
lifecycle/stale Stale type/bug
Dominant language
Java
Stars
15.3k
Forks
3.8k
Avg merge
1d 14h
Merged PRs (30d)
160

Description

**Describe the bug**
Using a standalone broker built at HEAD (as of 12/7/2021), attempting to delete topics a second or two after they are created returns a 500 error and a stacktrace.

This occurs primarily in tenant/namespace combos that are also created very shortly before the topic deletion is attempted (e.g. I see this in test code a lot).

This error is intermittent; it does not occur every time, and sometimes topic deletion is successful.

**To Reproduce**
1. Ensure a standalone broker is running.
2. Create a tenant, namespace, and persistent topic in that namespace. Do these creation operations back to back, milliseconds or less apart.
3. Attempt to delete that topic via the management API.
4. Do steps 2-3 repeatedly; the error should eventually occur.

**Expected behavior**
Topic deletion should always succeed.

**Desktop (please complete the following information):**
- OS: MacOS 11.6.1, pulsar client 2.8.1.

**Client stacktrace:**
```
E Response code: 500
E URL: DELETE http://127.0.0.1:8080/admin/v2/persistent/chariot-test-obj-Site-1-xUiJ-primary/chariot_namespace_chariot-test-obj-SynchronousChariotTopic-2-rKhX/chariot_topic_chariot-test-obj-SynchronousChariotTopic-2-rKhX?force=true
E Headers: Headers({'host': '127.0.0.1:8080', 'accept': '*/*', 'accept-encoding': 'gzip, deflate', 'connection': 'keep-alive', 'user-agent': 'python-httpx/0.20.0'})
E Payload: b''
E
E --- An unexpected error occurred in the server ---
E
E Message: org.apache.bookkeeper.mledger.ManagedLedgerException: java.util.concurrent.CompletionException: org.apache.bookkeeper.client.BKException$BKUnexpectedConditionException: Unexpected condition
E
E Stacktrace:
E
E java.util.concurrent.ExecutionException: org.apache.bookkeeper.mledger.ManagedLedgerException: java.util.concurrent.CompletionException: org.apache.bookkeeper.client.BKException$BKUnexpectedConditionException: Unexpected condition
E at java.base/java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:395)
E at java.base/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1999)
E at org.apache.pulsar.broker.admin.impl.PersistentTopicsBase.internalDeleteTopicForcefully(PersistentTopicsBase.java:304)
E at org.apache.pulsar.broker.admin.impl.PersistentTopicsBase.internalDeleteTopic(PersistentTopicsBase.java:1007)
E at org.apache.pulsar.broker.admin.v2.PersistentTopics.deleteTopic(PersistentTopics.java:894)
E at jdk.internal.reflect.GeneratedMethodAccessor225.invoke(Unknown Source)
E at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
E at java.base/java.lang.reflect.Method.invoke(Method.java:566)
E at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory.lambda$static$0(ResourceMethodInvocationHandlerFactory.java:52)
E at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:124)
E at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:167)
E at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$VoidOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:159)
E at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:79)
E at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:475)
E at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:397)
E at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:81)
E at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:255)
E at org.glassfish.jersey.internal.Errors$1.call(Errors.java:248)
E at org.glassfish.jersey.internal.Errors$1.call(Errors.java:244)
E at org.glassfish.jersey.internal.Errors.process(Errors.java:292)
E at org.glassfish.jersey.internal.Errors.process(Errors.java:274)
E at org.glassfish.jersey.internal.Errors.process(Errors.java:244)
E at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:265)
E at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:234)
E at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:680)
E at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:394)
E at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:346)
E at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:366)
E at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:319)
E at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:205)
E at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:799)
E at org.eclipse.jetty.servlet.ServletHandler$ChainEnd.doFilter(ServletHandler.java:1626)
E at org.apache.pulsar.broker.web.ResponseHandlerFilter.doFilter(ResponseHandlerFilter.java:67)
E at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
E at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601)
E at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:548)
E at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
E at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1624)
E at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
E at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1434)
E at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)
E at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:501)
E at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1594)
E at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)
E at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1349)
E at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
E at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:234)
E at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:146)
E at org.eclipse.jetty.server.handler.StatisticsHandler.handle(StatisticsHandler.java:179)
E at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
E at org.eclipse.jetty.server.Server.handle(Server.java:516)
E at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:388)
E at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:633)
E at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:380)
E at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:277)
E at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
E at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
E at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104)
E at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:338)
E at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:315)
E at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:173)
E at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131)
E at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:386)
E at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
E at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
E at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
E at java.base/java.lang.Thread.run(Thread.java:829)
E Caused by: org.apache.bookkeeper.mledger.ManagedLedgerException: java.util.concurrent.CompletionException: org.apache.bookkeeper.client.BKException$BKUnexpectedConditionException: Unexpected condition
E Caused by: java.util.concurrent.CompletionException: org.apache.bookkeeper.client.BKException$BKUnexpectedConditionException: Unexpected condition
E at java.base/java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:331)
E at java.base/java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:346)
E at java.base/java.util.concurrent.CompletableFuture$UniRun.tryFire(CompletableFuture.java:777)
E at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506)
E at java.base/java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2088)
E at org.apache.bookkeeper.mledger.util.Futures.lambda$waitForAll$0(Futures.java:57)
E at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:859)
E at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:837)
E at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506)
E at java.base/java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2088)
E at org.apache.bookkeeper.client.SyncCallbackUtils.finish(SyncCallbackUtils.java:83)
E at org.apache.bookkeeper.client.SyncCallbackUtils$SyncDeleteCallback.deleteComplete(SyncCallbackUtils.java:180)
E at org.apache.bookkeeper.client.LedgerDeleteOp.lambda$initiate$0(LedgerDeleteOp.java:87)
E at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:859)
E at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:837)
E at java.base/java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:478)
E at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
E at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
E at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
E at java.base/java.lang.Thread.run(Thread.java:829)
E Caused by: org.apache.bookkeeper.client.BKException$BKUnexpectedConditionException: Unexpected condition
E ... 10 more
```

Contributor guide

Open the contributing guide

Research direction

Reproduce the issue on a standalone broker by creating the tenant, namespace, and persistent topic back to back, then repeatedly deleting the topic through the management API. Start at PersistentTopicsBase.internalDeleteTopicForcefully and the BookKeeper LedgerDeleteOp path shown in the stack trace; done means deletion consistently succeeds without a 500 error or BKUnexpectedConditionException.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
backend, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.