Running Pinot with an invalid zookeeper service name seems to error during clean-up (in addition to connection error)
- Dominant language
- Java
- Stars
- 6.1k
- Forks
- 1.5k
- Avg merge
- 1d 21h
- Merged PRs (30d)
- 189
Description
Running with an invalid zookeeper service name seems to have an error during clean-up in addition to the main error, eg: `bin/pinot-admin.sh StartController -clusterName PinotCluster -controllerHost null -controllerPort 9000 -dataDir /tmp/PinotController -zkAddress pinotzookeeper.service:2181`
It appears that not only did it error on `java.net.UnknownHostException: pinotzookeeper.service: Name or service not known`, but it also error'd on afterwards (or during recovery).
```
java.lang.NullPointerException: null
at org.apache.helix.manager.zk.client.ZkConnectionManager.cleanupInactiveWatchers(ZkConnectionManager.java:112)
```
Looks like it just may not be skipping cleanup when no connection was established?
Here's the full stack trace.
```
Initiating client connection, connectString=pinotzookeeper.service:2181 sessionTimeout=30000 watcher=org.apache.helix.manager.zk.client.ZkConnectionManager@432038ec
Unable to connect to server: pinotzookeeper:2181
java.net.UnknownHostException: pinotzookeeper.service: Name or service not known
at java.net.Inet4AddressImpl.lookupAllHostAddr(Native Method) ~[?:1.8.0_252]
at java.net.InetAddress$2.lookupAllHostAddr(InetAddress.java:929) ~[?:1.8.0_252]
at java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1324) ~[?:1.8.0_252]
at java.net.InetAddress.getAllByName0(InetAddress.java:1277) ~[?:1.8.0_252]
at java.net.InetAddress.getAllByName(InetAddress.java:1193) ~[?:1.8.0_252]
at java.net.InetAddress.getAllByName(InetAddress.java:1127) ~[?:1.8.0_252]
at org.apache.zookeeper.client.StaticHostProvider.(StaticHostProvider.java:62) [pinot-all-0.5.0-SNAPSHOT-jar-with-dependencies.jar:0.5.0-SNAPSHOT-a8fbdaeffaf99c1df4258f60ecbbab1f553d7696]
at org.apache.zookeeper.ZooKeeper.(ZooKeeper.java:449) [pinot-all-0.5.0-SNAPSHOT-jar-with-dependencies.jar:0.5.0-SNAPSHOT-a8fbdaeffaf99c1df4258f60ecbbab1f553d7696]
at org.apache.zookeeper.ZooKeeper.(ZooKeeper.java:383) [pinot-all-0.5.0-SNAPSHOT-jar-with-dependencies.jar:0.5.0-SNAPSHOT-a8fbdaeffaf99c1df4258f60ecbbab1f553d7696]
at org.apache.helix.manager.zk.zookeeper.ZkConnection.connect(ZkConnection.java:63) [pinot-all-0.5.0-SNAPSHOT-jar-with-dependencies.jar:0.5.0-SNAPSHOT-a8fbdaeffaf99c1df4258f60ecbbab1f553d7696]
at org.apache.helix.manager.zk.zookeeper.ZkClient.connect(ZkClient.java:1565) [pinot-all-0.5.0-SNAPSHOT-jar-with-dependencies.jar:0.5.0-SNAPSHOT-a8fbdaeffaf99c1df4258f60ecbbab1f553d7696]
at org.apache.helix.manager.zk.zookeeper.ZkClient.(ZkClient.java:179) [pinot-all-0.5.0-SNAPSHOT-jar-with-dependencies.jar:0.5.0-SNAPSHOT-a8fbdaeffaf99c1df4258f60ecbbab1f553d7696]
at org.apache.helix.manager.zk.ZkClient.(ZkClient.java:87) [pinot-all-0.5.0-SNAPSHOT-jar-with-dependencies.jar:0.5.0-SNAPSHOT-a8fbdaeffaf99c1df4258f60ecbbab1f553d7696]
at org.apache.helix.manager.zk.client.ZkConnectionManager.(ZkConnectionManager.java:41) [pinot-all-0.5.0-SNAPSHOT-jar-with-dependencies.jar:0.5.0-SNAPSHOT-a8fbdaeffaf99c1df4258f60ecbbab1f553d7696]
at org.apache.helix.manager.zk.client.SharedZkClientFactory.getOrCreateZkConnectionNamanger(SharedZkClientFactory.java:60) [pinot-all-0.5.0-SNAPSHOT-jar-with-dependencies.jar:0.5.0-SNAPSHOT-a8fbdaeffaf99c1df4258f60ecbbab1f553d7696]
at org.apache.helix.manager.zk.client.SharedZkClientFactory.buildZkClient(SharedZkClientFactory.java:40) [pinot-all-0.5.0-SNAPSHOT-jar-with-dependencies.jar:0.5.0-SNAPSHOT-a8fbdaeffaf99c1df4258f60ecbbab1f553d7696]
at org.apache.helix.manager.zk.ZKHelixAdmin.(ZKHelixAdmin.java:107) [pinot-all-0.5.0-SNAPSHOT-jar-with-dependencies.jar:0.5.0-SNAPSHOT-a8fbdaeffaf99c1df4258f60ecbbab1f553d7696]
at org.apache.pinot.controller.helix.core.util.HelixSetupUtils.setupHelixClusterIfNeeded(HelixSetupUtils.java:69) [pinot-all-0.5.0-SNAPSHOT-jar-with-dependencies.jar:0.5.0-SNAPSHOT-a8fbdaeffaf99c1df4258f60ecbbab1f553d7696]
at org.apache.pinot.controller.helix.core.util.HelixSetupUtils.setupHelixController(HelixSetupUtils.java:63) [pinot-all-0.5.0-SNAPSHOT-jar-with-dependencies.jar:0.5.0-SNAPSHOT-a8fbdaeffaf99c1df4258f60ecbbab1f553d7696]
at org.apache.pinot.controller.ControllerStarter.setUpHelixController(ControllerStarter.java:310) [pinot-all-0.5.0-SNAPSHOT-jar-with-dependencies.jar:0.5.0-SNAPSHOT-a8fbdaeffaf99c1df4258f60ecbbab1f553d7696]
at org.apache.pinot.controller.ControllerStarter.start(ControllerStarter.java:289) [pinot-all-0.5.0-SNAPSHOT-jar-with-dependencies.jar:0.5.0-SNAPSHOT-a8fbdaeffaf99c1df4258f60ecbbab1f553d7696]
at org.apache.pinot.tools.service.PinotServiceManager.startController(PinotServiceManager.java:113) [pinot-all-0.5.0-SNAPSHOT-jar-with-dependencies.jar:0.5.0-SNAPSHOT-a8fbdaeffaf99c1df4258f60ecbbab1f553d7696]
at org.apache.pinot.tools.service.PinotServiceManager.startRole(PinotServiceManager.java:90) [pinot-all-0.5.0-SNAPSHOT-jar-with-dependencies.jar:0.5.0-SNAPSHOT-a8fbdaeffaf99c1df4258f60ecbbab1f553d7696]
at org.apache.pinot.tools.admin.command.StartServiceManagerCommand.startPinotService(StartServiceManagerCommand.java:200) [pinot-all-0.5.0-SNAPSHOT-jar-with-dependencies.jar:0.5.0-SNAPSHOT-a8fbdaeffaf99c1df4258f60ecbbab1f553d7696]
at org.apache.pinot.tools.admin.command.StartServiceManagerCommand.startPinotService(StartServiceManagerCommand.java:195) [pinot-all-0.5.0-SNAPSHOT-jar-with-dependencies.jar:0.5.0-SNAPSHOT-a8fbdaeffaf99c1df4258f60ecbbab1f553d7696]
at org.apache.pinot.tools.admin.command.StartServiceManagerCommand.execute(StartServiceManagerCommand.java:166) [pinot-all-0.5.0-SNAPSHOT-jar-with-dependencies.jar:0.5.0-SNAPSHOT-a8fbdaeffaf99c1df4258f60ecbbab1f553d7696]
at org.apache.pinot.tools.admin.command.StartControllerCommand.execute(StartControllerCommand.java:130) [pinot-all-0.5.0-SNAPSHOT-jar-with-dependencies.jar:0.5.0-SNAPSHOT-a8fbdaeffaf99c1df4258f60ecbbab1f553d7696]
at org.apache.pinot.tools.admin.PinotAdministrator.execute(PinotAdministrator.java:156) [pinot-all-0.5.0-SNAPSHOT-jar-with-dependencies.jar:0.5.0-SNAPSHOT-a8fbdaeffaf99c1df4258f60ecbbab1f553d7696]
at org.apache.pinot.tools.admin.PinotAdministrator.main(PinotAdministrator.java:168) [pinot-all-0.5.0-SNAPSHOT-jar-with-dependencies.jar:0.5.0-SNAPSHOT-a8fbdaeffaf99c1df4258f60ecbbab1f553d7696]
Failed to start a [ CONTROLLER ] Service
java.lang.NullPointerException: null
at org.apache.helix.manager.zk.client.ZkConnectionManager.cleanupInactiveWatchers(ZkConnectionManager.java:112) ~[pinot-all-0.5.0-SNAPSHOT-jar-with-dependencies.jar:0.5.0-SNAPSHOT-a8fbdaeffaf99c1df4258f60ecbbab1f553d7696]
at org.apache.helix.manager.zk.client.ZkConnectionManager.close(ZkConnectionManager.java:95) ~[pinot-all-0.5.0-SNAPSHOT-jar-with-dependencies.jar:0.5.0-SNAPSHOT-a8fbdaeffaf99c1df4258f60ecbbab1f553d7696]
at org.apache.helix.manager.zk.client.ZkConnectionManager.close(ZkConnectionManager.java:91) ~[pinot-all-0.5.0-SNAPSHOT-jar-with-dependencies.jar:0.5.0-SNAPSHOT-a8fbdaeffaf99c1df4258f60ecbbab1f553d7696]
at org.apache.helix.manager.zk.zookeeper.ZkClient.connect(ZkClient.java:1591) ~[pinot-all-0.5.0-SNAPSHOT-jar-with-dependencies.jar:0.5.0-SNAPSHOT-a8fbdaeffaf99c1df4258f60ecbbab1f553d7696]
at org.apache.helix.manager.zk.zookeeper.ZkClient.(ZkClient.java:179) ~[pinot-all-0.5.0-SNAPSHOT-jar-with-dependencies.jar:0.5.0-SNAPSHOT-a8fbdaeffaf99c1df4258f60ecbbab1f553d7696]
at org.apache.helix.manager.zk.ZkClient.(ZkClient.java:87) ~[pinot-all-0.5.0-SNAPSHOT-jar-with-dependencies.jar:0.5.0-SNAPSHOT-a8fbdaeffaf99c1df4258f60ecbbab1f553d7696]
at org.apache.helix.manager.zk.client.ZkConnectionManager.(ZkConnectionManager.java:41) ~[pinot-all-0.5.0-SNAPSHOT-jar-with-dependencies.jar:0.5.0-SNAPSHOT-a8fbdaeffaf99c1df4258f60ecbbab1f553d7696]
at org.apache.helix.manager.zk.client.SharedZkClientFactory.getOrCreateZkConnectionNamanger(SharedZkClientFactory.java:60) ~[pinot-all-0.5.0-SNAPSHOT-jar-with-dependencies.jar:0.5.0-SNAPSHOT-a8fbdaeffaf99c1df4258f60ecbbab1f553d7696]
at org.apache.helix.manager.zk.client.SharedZkClientFactory.buildZkClient(SharedZkClientFactory.java:40) ~[pinot-all-0.5.0-SNAPSHOT-jar-with-dependencies.jar:0.5.0-SNAPSHOT-a8fbdaeffaf99c1df4258f60ecbbab1f553d7696]
at org.apache.helix.manager.zk.ZKHelixAdmin.(ZKHelixAdmin.java:107) ~[pinot-all-0.5.0-SNAPSHOT-jar-with-dependencies.jar:0.5.0-SNAPSHOT-a8fbdaeffaf99c1df4258f60ecbbab1f553d7696]
at org.apache.pinot.controller.helix.core.util.HelixSetupUtils.setupHelixClusterIfNeeded(HelixSetupUtils.java:69) ~[pinot-all-0.5.0-SNAPSHOT-jar-with-dependencies.jar:0.5.0-SNAPSHOT-a8fbdaeffaf99c1df4258f60ecbbab1f553d7696]
at org.apache.pinot.controller.helix.core.util.HelixSetupUtils.setupHelixController(HelixSetupUtils.java:63) ~[pinot-all-0.5.0-SNAPSHOT-jar-with-dependencies.jar:0.5.0-SNAPSHOT-a8fbdaeffaf99c1df4258f60ecbbab1f553d7696]
at org.apache.pinot.controller.ControllerStarter.setUpHelixController(ControllerStarter.java:310) ~[pinot-all-0.5.0-SNAPSHOT-jar-with-dependencies.jar:0.5.0-SNAPSHOT-a8fbdaeffaf99c1df4258f60ecbbab1f553d7696]
at org.apache.pinot.controller.ControllerStarter.start(ControllerStarter.java:289) ~[pinot-all-0.5.0-SNAPSHOT-jar-with-dependencies.jar:0.5.0-SNAPSHOT-a8fbdaeffaf99c1df4258f60ecbbab1f553d7696]
at org.apache.pinot.tools.service.PinotServiceManager.startController(PinotServiceManager.java:113) ~[pinot-all-0.5.0-SNAPSHOT-jar-with-dependencies.jar:0.5.0-SNAPSHOT-a8fbdaeffaf99c1df4258f60ecbbab1f553d7696]
at org.apache.pinot.tools.service.PinotServiceManager.startRole(PinotServiceManager.java:90) ~[pinot-all-0.5.0-SNAPSHOT-jar-with-dependencies.jar:0.5.0-SNAPSHOT-a8fbdaeffaf99c1df4258f60ecbbab1f553d7696]
at org.apache.pinot.tools.admin.command.StartServiceManagerCommand.startPinotService(StartServiceManagerCommand.java:200) [pinot-all-0.5.0-SNAPSHOT-jar-with-dependencies.jar:0.5.0-SNAPSHOT-a8fbdaeffaf99c1df4258f60ecbbab1f553d7696]
at org.apache.pinot.tools.admin.command.StartServiceManagerCommand.startPinotService(StartServiceManagerCommand.java:195) [pinot-all-0.5.0-SNAPSHOT-jar-with-dependencies.jar:0.5.0-SNAPSHOT-a8fbdaeffaf99c1df4258f60ecbbab1f553d7696]
at org.apache.pinot.tools.admin.command.StartServiceManagerCommand.execute(StartServiceManagerCommand.java:166) [pinot-all-0.5.0-SNAPSHOT-jar-with-dependencies.jar:0.5.0-SNAPSHOT-a8fbdaeffaf99c1df4258f60ecbbab1f553d7696]
at org.apache.pinot.tools.admin.command.StartControllerCommand.execute(StartControllerCommand.java:130) [pinot-all-0.5.0-SNAPSHOT-jar-with-dependencies.jar:0.5.0-SNAPSHOT-a8fbdaeffaf99c1df4258f60ecbbab1f553d7696]
at org.apache.pinot.tools.admin.PinotAdministrator.execute(PinotAdministrator.java:156) [pinot-all-0.5.0-SNAPSHOT-jar-with-dependencies.jar:0.5.0-SNAPSHOT-a8fbdaeffaf99c1df4258f60ecbbab1f553d7696]
at org.apache.pinot.tools.admin.PinotAdministrator.main(PinotAdministrator.java:168) [pinot-all-0.5.0-SNAPSHOT-jar-with-dependencies.jar:0.5.0-SNAPSHOT-a8fbdaeffaf99c1df4258f60ecbbab1f553d7696]
```
Contributor guide
Research direction
Start with the ZkConnectionManager cleanupInactiveWatchers() and close() frames in the stack trace, then follow the connection setup from HelixSetupUtils.setupHelixClusterIfNeeded() and ControllerStarter.setUpHelixController(). Reproduce the controller start with the invalid -zkAddress shown in the issue. Done means the original connection failure remains clear without a secondary cleanup NullPointerException.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- databases, distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 30/100