apache / apache/pulsar

Attempting to delete a tenant that was recently created sometimes fails with "MetadataStoreException: org.apache.zookeeper.KeeperException$NotEmptyException: KeeperErrorCode = Directory not empty for /admin/policies/$tenantname"

Open
#12,553 8 comments 0 reactions 0 assignees View on GitHub
type/bug
Dominant language
Java
Stars
15.3k
Forks
3.8k
Avg merge
1d 14h
Merged PRs (30d)
160

Description

**Describe the bug**
When I create a tenant/namespace/topic/subscription and then immediately (seconds or less) try to delete them, I sometimes get a 500 response code and an internal stacktrace (see below).

**To Reproduce**
Run reproduction plan for https://github.com/apache/pulsar/issues/12551; sometimes no error will occur, sometimes the error described in that issue or others will occur, and sometimes this error will occur.

**Expected behavior**
1. Tenant deletion should succeed if previous tenant creation succeeded, no matter how recently previous tenant creation was performed.
2. This is probably more important: all operations related to deletion/creation of topics/namespaces/tenants conclude *before* the management API returns success codes to the user. This bug and the other similar ones I filed (see github links below) all seem to arise from CRUD operations with the management API being asynchronous: i.e. when I create/delete a tenant/topic/namespace, the actual side effects of that creation or deletion (e.g. adding/removing ledgers in BookKeeper, updating metadata in ZK) occur later, not during the API post. Not only is that bound to cause bugs like this, but it's also not what users expect; I would be happy to wait seconds or minutes for management API operations to complete in exchange for knowing that when they successfully complete that the thing I requested has actually been done.

**Environment**
Same environment as https://github.com/apache/pulsar/issues/12551

**Stacktrace**
```
URL: DELETE http://pulsar-blt-sn-platform-proxy-headless.pulsar:8080/admin/v2/tenants/blt3
Headers: Headers({'host': 'pulsar-blt-sn-platform-proxy-headless.pulsar:8080', 'accept': '*/*', 'accept-encoding': 'gzip, deflate', 'connection': 'keep-alive', 'user-agent': 'python-httpx/0.20.0'})
Payload: b''
--- An unexpected error occurred in the server ---

Message: org.apache.pulsar.metadata.api.MetadataStoreException: org.apache.zookeeper.KeeperException$NotEmptyException: KeeperErrorCode = Directory not empty for /admin/policies/blt3

Stacktrace:

java.util.concurrent.CompletionException: org.apache.pulsar.metadata.api.MetadataStoreException: org.apache.zookeeper.KeeperException$NotEmptyException: KeeperErrorCode = Directory not empty for /admin/policies/blt3
at java.base/java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:331)
at java.base/java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:346)
at java.base/java.util.concurrent.CompletableFuture$UniRun.tryFire(CompletableFuture.java:777)
at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506)
at java.base/java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2088)
at org.apache.pulsar.metadata.impl.ZKMetadataStore.lambda$null$21(ZKMetadataStore.java:277)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: org.apache.pulsar.metadata.api.MetadataStoreException: org.apache.zookeeper.KeeperException$NotEmptyException: KeeperErrorCode = Directory not empty for /admin/policies/blt3
at org.apache.pulsar.metadata.impl.ZKMetadataStore.getException(ZKMetadataStore.java:314)
... 5 more
Caused by: org.apache.zookeeper.KeeperException$NotEmptyException: KeeperErrorCode = Directory not empty for /admin/policies/blt3
at org.apache.zookeeper.KeeperException.create(KeeperException.java:132)
at org.apache.zookeeper.KeeperException.create(KeeperException.java:54)
at org.apache.pulsar.metadata.impl.ZKMetadataStore.getException(ZKMetadataStore.java:304)
... 5 more
```

Contributor guide

Open the contributing guide

Research direction

Start with the DELETE tenant reproduction described here and the linked reproduction plan in issue 12551. Inspect org.apache.pulsar.metadata.impl.ZKMetadataStore, especially the stack-trace locations in ZKMetadataStore.java:277 and :314, to trace the NotEmptyException. Done should mean recently created tenants can be deleted successfully and the management operation's completion matches the expected deletion side effects.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
backend-api-design, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.