Setting topic permission under load

#### Expected behavior

Do a REST call to admin interface for setting topic permissions like:

https://localhost:8081/admin/v2/persistent/tt1/testns1/t4%5Cdata/permissions/testrole

body:

{'testrole': ['consume', 'produce']}

Should work every time and return status 204.

#### Actual behavior

When this is done with heavy load setting up permission for many topics in parallel. Some will result in the below logged exception and status code 500.

#### Steps to reproduce

Set up tenant and namespace, then run the set permission for many different topics in that namespace in parallel. I use the same role and actions for all of them.

#### System configuration
pulsar: 2.2.0

This error was when running the broker in standalone mode in a docker container supplied on docker hub, with authorisation and authentication via client certs. Have not tested on our full pulsar setup yet.

#### Logs

12:27:31.613 [pulsar-web-53-28] INFO org.apache.pulsar.broker.admin.impl.PersistentTopicsBase - [superUser] Successfully granted access for role testrole: [produce, consume] - topic persistent://tt1/testns1/t5\meta
12:27:31.587 [pulsar-web-53-30] ERROR org.apache.pulsar.broker.admin.impl.PersistentTopicsBase - [superUser] Failed to grant permissions for topic persistent://tt1/testns1/t0\data
org.apache.zookeeper.KeeperException$BadVersionException: KeeperErrorCode = BadVersion for /admin/policies/tt1/testns1
at org.apache.zookeeper.KeeperException.create(KeeperException.java:118) ~[org.apache.pulsar-pulsar-zookeeper-2.2.0.jar:2.2.0]
at org.apache.zookeeper.KeeperException.create(KeeperException.java:54) ~[org.apache.pulsar-pulsar-zookeeper-2.2.0.jar:2.2.0]
at org.apache.zookeeper.ZooKeeper.setData(ZooKeeper.java:1336) ~[org.apache.pulsar-pulsar-zookeeper-2.2.0.jar:2.2.0]
at org.apache.bookkeeper.zookeeper.ZooKeeperClient.access$3101(ZooKeeperClient.java:70) ~[org.apache.bookkeeper-bookkeeper-server-4.7.2.jar:4.7.2]
at org.apache.bookkeeper.zookeeper.ZooKeeperClient$21.call(ZooKeeperClient.java:1065) ~[org.apache.bookkeeper-bookkeeper-server-4.7.2.jar:4.7.2]
at org.apache.bookkeeper.zookeeper.ZooKeeperClient$21.call(ZooKeeperClient.java:1059) ~[org.apache.bookkeeper-bookkeeper-server-4.7.2.jar:4.7.2]
at org.apache.bookkeeper.zookeeper.ZooWorker.syncCallWithRetries(ZooWorker.java:140) ~[org.apache.bookkeeper-bookkeeper-server-4.7.2.jar:4.7.2]
at org.apache.bookkeeper.zookeeper.ZooKeeperClient.setData(ZooKeeperClient.java:1059) ~[org.apache.bookkeeper-bookkeeper-server-4.7.2.jar:4.7.2]
at org.apache.pulsar.broker.admin.impl.PersistentTopicsBase.internalGrantPermissionsOnTopic(PersistentTopicsBase.java:274) [org.apache.pulsar-pulsar-broker-2.2.0.jar:2.2.0]
at org.apache.pulsar.broker.admin.v2.PersistentTopics.grantPermissionsOnTopic(PersistentTopics.java:107) [org.apache.pulsar-pulsar-broker-2.2.0.jar:2.2.0]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_171]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_171]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_171]
at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_171]
at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory$1.invoke(ResourceMethodInvocationHandlerFactory.java:81) [org.glassfish.jersey.core-jersey-server-2.25.jar:?]
at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:144) [org.glassfish.jersey.core-jersey-server-2.25.jar:?]
at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:161) [org.glassfish.jersey.core-jersey-server-2.25.jar:?]
at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$VoidOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:143) [org.glassfish.jersey.core-jersey-server-2.25.jar:?]
at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:99) [org.glassfish.jersey.core-jersey-server-2.25.jar:?]
at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:389) [org.glassfish.jersey.core-jersey-server-2.25.jar:?]
at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:347) [org.glassfish.jersey.core-jersey-server-2.25.jar:?]
at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:102) [org.glassfish.jersey.core-jersey-server-2.25.jar:?]
at org.glassfish.jersey.server.ServerRuntime$2.run(ServerRuntime.java:326) [org.glassfish.jersey.core-jersey-server-2.25.jar:?]
at org.glassfish.jersey.internal.Errors$1.call(Errors.java:271) [org.glassfish.jersey.core-jersey-common-2.25.jar:?]
at org.glassfish.jersey.internal.Errors$1.call(Errors.java:267) [org.glassfish.jersey.core-jersey-common-2.25.jar:?]
at org.glassfish.jersey.internal.Errors.process(Errors.java:315) [org.glassfish.jersey.core-jersey-common-2.25.jar:?]
at org.glassfish.jersey.internal.Errors.process(Errors.java:297) [org.glassfish.jersey.core-jersey-common-2.25.jar:?]
at org.glassfish.jersey.internal.Errors.process(Errors.java:267) [org.glassfish.jersey.core-jersey-common-2.25.jar:?]
at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:317) [org.glassfish.jersey.core-jersey-common-2.25.jar:?]
at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:305) [org.glassfish.jersey.core-jersey-server-2.25.jar:?]
at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:1154) [org.glassfish.jersey.core-jersey-server-2.25.jar:?]
at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:473) [org.glassfish.jersey.containers-jersey-container-servlet-core-2.25.jar:?]
at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:427) [org.glassfish.jersey.containers-jersey-container-servlet-core-2.25.jar:?]
at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:388) [org.glassfish.jersey.containers-jersey-container-servlet-core-2.25.jar:?]
at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:341) [org.glassfish.jersey.containers-jersey-container-servlet-core-2.25.jar:?]
at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:228) [org.glassfish.jersey.containers-jersey-container-servlet-core-2.25.jar:?]
at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:845) [org.eclipse.jetty-jetty-servlet-9.3.11.v20160721.jar:9.3.11.v20160721]
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1689) [org.eclipse.jetty-jetty-servlet-9.3.11.v20160721.jar:9.3.11.v20160721]
at org.apache.pulsar.broker.web.ResponseHandlerFilter.doFilter(ResponseHandlerFilter.java:53) [org.apache.pulsar-pulsar-broker-2.2.0.jar:2.2.0]
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1676) [org.eclipse.jetty-jetty-servlet-9.3.11.v20160721.jar:9.3.11.v20160721]
at org.apache.pulsar.broker.web.AuthenticationFilter.doFilter(AuthenticationFilter.java:74) [org.apache.pulsar-pulsar-broker-common-2.2.0.jar:2.2.0]
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1676) [org.eclipse.jetty-jetty-servlet-9.3.11.v20160721.jar:9.3.11.v20160721]
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:581) [org.eclipse.jetty-jetty-servlet-9.3.11.v20160721.jar:9.3.11.v20160721]
at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:224) [org.eclipse.jetty-jetty-server-9.3.11.v20160721.jar:9.3.11.v20160721]
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180) [org.eclipse.jetty-jetty-server-9.3.11.v20160721.jar:9.3.11.v20160721]
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:511) [org.eclipse.jetty-jetty-servlet-9.3.11.v20160721.jar:9.3.11.v20160721]
at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185) [org.eclipse.jetty-jetty-server-9.3.11.v20160721.jar:9.3.11.v20160721]
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112) [org.eclipse.jetty-jetty-server-9.3.11.v20160721.jar:9.3.11.v20160721]
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) [org.eclipse.jetty-jetty-server-9.3.11.v20160721.jar:9.3.11.v20160721]
at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:213) [org.eclipse.jetty-jetty-server-9.3.11.v20160721.jar:9.3.11.v20160721]
at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:119) [org.eclipse.jetty-jetty-server-9.3.11.v20160721.jar:9.3.11.v20160721]
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134) [org.eclipse.jetty-jetty-server-9.3.11.v20160721.jar:9.3.11.v20160721]
at org.eclipse.jetty.server.Server.handle(Server.java:524) [org.eclipse.jetty-jetty-server-9.3.11.v20160721.jar:9.3.11.v20160721]
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:319) [org.eclipse.jetty-jetty-server-9.3.11.v20160721.jar:9.3.11.v20160721]
at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:253) [org.eclipse.jetty-jetty-server-9.3.11.v20160721.jar:9.3.11.v20160721]
at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273) [org.eclipse.jetty-jetty-io-9.3.11.v20160721.jar:9.3.11.v20160721]
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95) [org.eclipse.jetty-jetty-io-9.3.11.v20160721.jar:9.3.11.v20160721]
at org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:202) [org.eclipse.jetty-jetty-io-9.3.11.v20160721.jar:9.3.11.v20160721]
at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273) [org.eclipse.jetty-jetty-io-9.3.11.v20160721.jar:9.3.11.v20160721]
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95) [org.eclipse.jetty-jetty-io-9.3.11.v20160721.jar:9.3.11.v20160721]
at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93) [org.eclipse.jetty-jetty-io-9.3.11.v20160721.jar:9.3.11.v20160721]
at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303) [org.apache.pulsar-pulsar-functions-metrics-2.2.0.jar:?]
at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148) [org.apache.pulsar-pulsar-functions-metrics-2.2.0.jar:?]
at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136) [org.apache.pulsar-pulsar-functions-metrics-2.2.0.jar:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_171]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_171]
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [io.netty-netty-all-4.1.22.Final.jar:4.1.22.Final]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_171]
12:27:31.669 [pulsar-web-53-27] INFO org.eclipse.jetty.server.RequestLog - 172.17.0.1 - - [07/Nov/2018:12:27:31 +0000] "POST https://localhost:8081/admin/v2/persistent/tt1/testns1/t6%5Cdata/permissions/testrole HTTP/1.1" 500 6238 "-" "Python/3.7 aiohttp/3.4.2" 385
12:27:31.669 [pulsar-web-53-31] INFO org.eclipse.jetty.server.RequestLog - 172.17.0.1 - - [07/Nov/2018:12:27:31 +0000] "POST https://localhost:8081/admin/v2/persistent/tt1/testns1/t7%5Cmeta/permissions/testrole HTTP/1.1" 500 5652 "-" "Python/3.7 aiohttp/3.4.2" 402
12:27:31.668 [pulsar-web-53-29] INFO org.eclipse.jetty.server.RequestLog - 172.17.0.1 - - [07/Nov/2018:12:27:31 +0000] "POST https://localhost:8081/admin/v2/persistent/tt1/testns1/t8%5Cdata/permissions/testrole HTTP/1.1" 500 6238 "-" "Python/3.7 aiohttp/3.4.2" 371
12:27:31.643 [pulsar-web-53-28] INFO org.eclipse.jetty.server.RequestLog - 172.17.0.1 - - [07/Nov/2018:12:27:31 +0000] "POST https://localhost:8081/admin/v2/persistent/tt1/testns1/t5%5Cmeta/permissions/testrole HTTP/1.1" 204 0 "-" "Python/3.7 aiohttp/3.4.2" 173
12:27:31.642 [pulsar-web-53-24] INFO org.eclipse.jetty.server.RequestLog - 172.17.0.1 - - [07/Nov/2018:12:27:31 +0000] "POST https://localhost:8081/admin/v2/persistent/tt1/testns1/t4%5Cdata/permissions/testrole HTTP/1.1" 500 6238 "-" "Python/3.7 aiohttp/3.4.2" 341

Start with PersistentTopicsBase.java:274 and PersistentTopics.java:107, then trace the ZooKeeper setData call that updates /admin/policies/{tenant}/{namespace}. Reproduce parallel permission requests for many topics and confirm that concurrent updates no longer return 500 or KeeperException.BadVersionException, while successful requests return 204.

Written by the indexing model from the issue text.