apache / apache/pinot

Running Pinot with an invalid zookeeper service name seems to error during clean-up (in addition to connection error)

Open
#5,802 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
6.1k
Forks
1.5k
Avg merge
1d 21h
Merged PRs (30d)
189

Description

Running with an invalid zookeeper service name seems to have an error during clean-up in addition to the main error, eg: `bin/pinot-admin.sh StartController -clusterName PinotCluster -controllerHost null -controllerPort 9000 -dataDir /tmp/PinotController -zkAddress pinotzookeeper.service:2181`

It appears that not only did it error on `java.net.UnknownHostException: pinotzookeeper.service: Name or service not known`, but it also error'd on afterwards (or during recovery).
```
java.lang.NullPointerException: null
at org.apache.helix.manager.zk.client.ZkConnectionManager.cleanupInactiveWatchers(ZkConnectionManager.java:112)
```

Looks like it just may not be skipping cleanup when no connection was established?

Here's the full stack trace.

```
Initiating client connection, connectString=pinotzookeeper.service:2181 sessionTimeout=30000 watcher=org.apache.helix.manager.zk.client.ZkConnectionManager@432038ec
Unable to connect to server: pinotzookeeper:2181
java.net.UnknownHostException: pinotzookeeper.service: Name or service not known
at java.net.Inet4AddressImpl.lookupAllHostAddr(Native Method) ~[?:1.8.0_252]
at java.net.InetAddress$2.lookupAllHostAddr(InetAddress.java:929) ~[?:1.8.0_252]
at java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1324) ~[?:1.8.0_252]
at java.net.InetAddress.getAllByName0(InetAddress.java:1277) ~[?:1.8.0_252]
at java.net.InetAddress.getAllByName(InetAddress.java:1193) ~[?:1.8.0_252]
at java.net.InetAddress.getAllByName(InetAddress.java:1127) ~[?:1.8.0_252]
at org.apache.zookeeper.client.StaticHostProvider.(StaticHostProvider.java:62) [pinot-all-0.5.0-SNAPSHOT-jar-with-dependencies.jar:0.5.0-SNAPSHOT-a8fbdaeffaf99c1df4258f60ecbbab1f553d7696]
at org.apache.zookeeper.ZooKeeper.(ZooKeeper.java:449) [pinot-all-0.5.0-SNAPSHOT-jar-with-dependencies.jar:0.5.0-SNAPSHOT-a8fbdaeffaf99c1df4258f60ecbbab1f553d7696]
at org.apache.zookeeper.ZooKeeper.(ZooKeeper.java:383) [pinot-all-0.5.0-SNAPSHOT-jar-with-dependencies.jar:0.5.0-SNAPSHOT-a8fbdaeffaf99c1df4258f60ecbbab1f553d7696]
at org.apache.helix.manager.zk.zookeeper.ZkConnection.connect(ZkConnection.java:63) [pinot-all-0.5.0-SNAPSHOT-jar-with-dependencies.jar:0.5.0-SNAPSHOT-a8fbdaeffaf99c1df4258f60ecbbab1f553d7696]
at org.apache.helix.manager.zk.zookeeper.ZkClient.connect(ZkClient.java:1565) [pinot-all-0.5.0-SNAPSHOT-jar-with-dependencies.jar:0.5.0-SNAPSHOT-a8fbdaeffaf99c1df4258f60ecbbab1f553d7696]
at org.apache.helix.manager.zk.zookeeper.ZkClient.(ZkClient.java:179) [pinot-all-0.5.0-SNAPSHOT-jar-with-dependencies.jar:0.5.0-SNAPSHOT-a8fbdaeffaf99c1df4258f60ecbbab1f553d7696]
at org.apache.helix.manager.zk.ZkClient.(ZkClient.java:87) [pinot-all-0.5.0-SNAPSHOT-jar-with-dependencies.jar:0.5.0-SNAPSHOT-a8fbdaeffaf99c1df4258f60ecbbab1f553d7696]
at org.apache.helix.manager.zk.client.ZkConnectionManager.(ZkConnectionManager.java:41) [pinot-all-0.5.0-SNAPSHOT-jar-with-dependencies.jar:0.5.0-SNAPSHOT-a8fbdaeffaf99c1df4258f60ecbbab1f553d7696]
at org.apache.helix.manager.zk.client.SharedZkClientFactory.getOrCreateZkConnectionNamanger(SharedZkClientFactory.java:60) [pinot-all-0.5.0-SNAPSHOT-jar-with-dependencies.jar:0.5.0-SNAPSHOT-a8fbdaeffaf99c1df4258f60ecbbab1f553d7696]
at org.apache.helix.manager.zk.client.SharedZkClientFactory.buildZkClient(SharedZkClientFactory.java:40) [pinot-all-0.5.0-SNAPSHOT-jar-with-dependencies.jar:0.5.0-SNAPSHOT-a8fbdaeffaf99c1df4258f60ecbbab1f553d7696]
at org.apache.helix.manager.zk.ZKHelixAdmin.(ZKHelixAdmin.java:107) [pinot-all-0.5.0-SNAPSHOT-jar-with-dependencies.jar:0.5.0-SNAPSHOT-a8fbdaeffaf99c1df4258f60ecbbab1f553d7696]
at org.apache.pinot.controller.helix.core.util.HelixSetupUtils.setupHelixClusterIfNeeded(HelixSetupUtils.java:69) [pinot-all-0.5.0-SNAPSHOT-jar-with-dependencies.jar:0.5.0-SNAPSHOT-a8fbdaeffaf99c1df4258f60ecbbab1f553d7696]
at org.apache.pinot.controller.helix.core.util.HelixSetupUtils.setupHelixController(HelixSetupUtils.java:63) [pinot-all-0.5.0-SNAPSHOT-jar-with-dependencies.jar:0.5.0-SNAPSHOT-a8fbdaeffaf99c1df4258f60ecbbab1f553d7696]
at org.apache.pinot.controller.ControllerStarter.setUpHelixController(ControllerStarter.java:310) [pinot-all-0.5.0-SNAPSHOT-jar-with-dependencies.jar:0.5.0-SNAPSHOT-a8fbdaeffaf99c1df4258f60ecbbab1f553d7696]
at org.apache.pinot.controller.ControllerStarter.start(ControllerStarter.java:289) [pinot-all-0.5.0-SNAPSHOT-jar-with-dependencies.jar:0.5.0-SNAPSHOT-a8fbdaeffaf99c1df4258f60ecbbab1f553d7696]
at org.apache.pinot.tools.service.PinotServiceManager.startController(PinotServiceManager.java:113) [pinot-all-0.5.0-SNAPSHOT-jar-with-dependencies.jar:0.5.0-SNAPSHOT-a8fbdaeffaf99c1df4258f60ecbbab1f553d7696]
at org.apache.pinot.tools.service.PinotServiceManager.startRole(PinotServiceManager.java:90) [pinot-all-0.5.0-SNAPSHOT-jar-with-dependencies.jar:0.5.0-SNAPSHOT-a8fbdaeffaf99c1df4258f60ecbbab1f553d7696]
at org.apache.pinot.tools.admin.command.StartServiceManagerCommand.startPinotService(StartServiceManagerCommand.java:200) [pinot-all-0.5.0-SNAPSHOT-jar-with-dependencies.jar:0.5.0-SNAPSHOT-a8fbdaeffaf99c1df4258f60ecbbab1f553d7696]
at org.apache.pinot.tools.admin.command.StartServiceManagerCommand.startPinotService(StartServiceManagerCommand.java:195) [pinot-all-0.5.0-SNAPSHOT-jar-with-dependencies.jar:0.5.0-SNAPSHOT-a8fbdaeffaf99c1df4258f60ecbbab1f553d7696]
at org.apache.pinot.tools.admin.command.StartServiceManagerCommand.execute(StartServiceManagerCommand.java:166) [pinot-all-0.5.0-SNAPSHOT-jar-with-dependencies.jar:0.5.0-SNAPSHOT-a8fbdaeffaf99c1df4258f60ecbbab1f553d7696]
at org.apache.pinot.tools.admin.command.StartControllerCommand.execute(StartControllerCommand.java:130) [pinot-all-0.5.0-SNAPSHOT-jar-with-dependencies.jar:0.5.0-SNAPSHOT-a8fbdaeffaf99c1df4258f60ecbbab1f553d7696]
at org.apache.pinot.tools.admin.PinotAdministrator.execute(PinotAdministrator.java:156) [pinot-all-0.5.0-SNAPSHOT-jar-with-dependencies.jar:0.5.0-SNAPSHOT-a8fbdaeffaf99c1df4258f60ecbbab1f553d7696]
at org.apache.pinot.tools.admin.PinotAdministrator.main(PinotAdministrator.java:168) [pinot-all-0.5.0-SNAPSHOT-jar-with-dependencies.jar:0.5.0-SNAPSHOT-a8fbdaeffaf99c1df4258f60ecbbab1f553d7696]
Failed to start a [ CONTROLLER ] Service
java.lang.NullPointerException: null
at org.apache.helix.manager.zk.client.ZkConnectionManager.cleanupInactiveWatchers(ZkConnectionManager.java:112) ~[pinot-all-0.5.0-SNAPSHOT-jar-with-dependencies.jar:0.5.0-SNAPSHOT-a8fbdaeffaf99c1df4258f60ecbbab1f553d7696]
at org.apache.helix.manager.zk.client.ZkConnectionManager.close(ZkConnectionManager.java:95) ~[pinot-all-0.5.0-SNAPSHOT-jar-with-dependencies.jar:0.5.0-SNAPSHOT-a8fbdaeffaf99c1df4258f60ecbbab1f553d7696]
at org.apache.helix.manager.zk.client.ZkConnectionManager.close(ZkConnectionManager.java:91) ~[pinot-all-0.5.0-SNAPSHOT-jar-with-dependencies.jar:0.5.0-SNAPSHOT-a8fbdaeffaf99c1df4258f60ecbbab1f553d7696]
at org.apache.helix.manager.zk.zookeeper.ZkClient.connect(ZkClient.java:1591) ~[pinot-all-0.5.0-SNAPSHOT-jar-with-dependencies.jar:0.5.0-SNAPSHOT-a8fbdaeffaf99c1df4258f60ecbbab1f553d7696]
at org.apache.helix.manager.zk.zookeeper.ZkClient.(ZkClient.java:179) ~[pinot-all-0.5.0-SNAPSHOT-jar-with-dependencies.jar:0.5.0-SNAPSHOT-a8fbdaeffaf99c1df4258f60ecbbab1f553d7696]
at org.apache.helix.manager.zk.ZkClient.(ZkClient.java:87) ~[pinot-all-0.5.0-SNAPSHOT-jar-with-dependencies.jar:0.5.0-SNAPSHOT-a8fbdaeffaf99c1df4258f60ecbbab1f553d7696]
at org.apache.helix.manager.zk.client.ZkConnectionManager.(ZkConnectionManager.java:41) ~[pinot-all-0.5.0-SNAPSHOT-jar-with-dependencies.jar:0.5.0-SNAPSHOT-a8fbdaeffaf99c1df4258f60ecbbab1f553d7696]
at org.apache.helix.manager.zk.client.SharedZkClientFactory.getOrCreateZkConnectionNamanger(SharedZkClientFactory.java:60) ~[pinot-all-0.5.0-SNAPSHOT-jar-with-dependencies.jar:0.5.0-SNAPSHOT-a8fbdaeffaf99c1df4258f60ecbbab1f553d7696]
at org.apache.helix.manager.zk.client.SharedZkClientFactory.buildZkClient(SharedZkClientFactory.java:40) ~[pinot-all-0.5.0-SNAPSHOT-jar-with-dependencies.jar:0.5.0-SNAPSHOT-a8fbdaeffaf99c1df4258f60ecbbab1f553d7696]
at org.apache.helix.manager.zk.ZKHelixAdmin.(ZKHelixAdmin.java:107) ~[pinot-all-0.5.0-SNAPSHOT-jar-with-dependencies.jar:0.5.0-SNAPSHOT-a8fbdaeffaf99c1df4258f60ecbbab1f553d7696]
at org.apache.pinot.controller.helix.core.util.HelixSetupUtils.setupHelixClusterIfNeeded(HelixSetupUtils.java:69) ~[pinot-all-0.5.0-SNAPSHOT-jar-with-dependencies.jar:0.5.0-SNAPSHOT-a8fbdaeffaf99c1df4258f60ecbbab1f553d7696]
at org.apache.pinot.controller.helix.core.util.HelixSetupUtils.setupHelixController(HelixSetupUtils.java:63) ~[pinot-all-0.5.0-SNAPSHOT-jar-with-dependencies.jar:0.5.0-SNAPSHOT-a8fbdaeffaf99c1df4258f60ecbbab1f553d7696]
at org.apache.pinot.controller.ControllerStarter.setUpHelixController(ControllerStarter.java:310) ~[pinot-all-0.5.0-SNAPSHOT-jar-with-dependencies.jar:0.5.0-SNAPSHOT-a8fbdaeffaf99c1df4258f60ecbbab1f553d7696]
at org.apache.pinot.controller.ControllerStarter.start(ControllerStarter.java:289) ~[pinot-all-0.5.0-SNAPSHOT-jar-with-dependencies.jar:0.5.0-SNAPSHOT-a8fbdaeffaf99c1df4258f60ecbbab1f553d7696]
at org.apache.pinot.tools.service.PinotServiceManager.startController(PinotServiceManager.java:113) ~[pinot-all-0.5.0-SNAPSHOT-jar-with-dependencies.jar:0.5.0-SNAPSHOT-a8fbdaeffaf99c1df4258f60ecbbab1f553d7696]
at org.apache.pinot.tools.service.PinotServiceManager.startRole(PinotServiceManager.java:90) ~[pinot-all-0.5.0-SNAPSHOT-jar-with-dependencies.jar:0.5.0-SNAPSHOT-a8fbdaeffaf99c1df4258f60ecbbab1f553d7696]
at org.apache.pinot.tools.admin.command.StartServiceManagerCommand.startPinotService(StartServiceManagerCommand.java:200) [pinot-all-0.5.0-SNAPSHOT-jar-with-dependencies.jar:0.5.0-SNAPSHOT-a8fbdaeffaf99c1df4258f60ecbbab1f553d7696]
at org.apache.pinot.tools.admin.command.StartServiceManagerCommand.startPinotService(StartServiceManagerCommand.java:195) [pinot-all-0.5.0-SNAPSHOT-jar-with-dependencies.jar:0.5.0-SNAPSHOT-a8fbdaeffaf99c1df4258f60ecbbab1f553d7696]
at org.apache.pinot.tools.admin.command.StartServiceManagerCommand.execute(StartServiceManagerCommand.java:166) [pinot-all-0.5.0-SNAPSHOT-jar-with-dependencies.jar:0.5.0-SNAPSHOT-a8fbdaeffaf99c1df4258f60ecbbab1f553d7696]
at org.apache.pinot.tools.admin.command.StartControllerCommand.execute(StartControllerCommand.java:130) [pinot-all-0.5.0-SNAPSHOT-jar-with-dependencies.jar:0.5.0-SNAPSHOT-a8fbdaeffaf99c1df4258f60ecbbab1f553d7696]
at org.apache.pinot.tools.admin.PinotAdministrator.execute(PinotAdministrator.java:156) [pinot-all-0.5.0-SNAPSHOT-jar-with-dependencies.jar:0.5.0-SNAPSHOT-a8fbdaeffaf99c1df4258f60ecbbab1f553d7696]
at org.apache.pinot.tools.admin.PinotAdministrator.main(PinotAdministrator.java:168) [pinot-all-0.5.0-SNAPSHOT-jar-with-dependencies.jar:0.5.0-SNAPSHOT-a8fbdaeffaf99c1df4258f60ecbbab1f553d7696]
```

Contributor guide

Open the contributing guide

Research direction

Start with the ZkConnectionManager cleanupInactiveWatchers() and close() frames in the stack trace, then follow the connection setup from HelixSetupUtils.setupHelixClusterIfNeeded() and ControllerStarter.setUpHelixController(). Reproduce the controller start with the invalid -zkAddress shown in the issue. Done means the original connection failure remains clear without a secondary cleanup NullPointerException.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
databases, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.