apache / apache/pinot

bug: broker crash at startup with `zip file closed`

Open
#6,669 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
6.1k
Forks
1.5k
Avg merge
1d 21h
Merged PRs (30d)
189

Description

I'm trying to up a pinot cluster for my tests with the following docker-compose configuration:

```yaml
version: '3'
services:
zookeeper:
image: zookeeper:3.5.6

pinot:
image: apachepinot/pinot:0.6.0
command: StartServiceManager -clusterName e2e -port 9001 -zkAddress zookeeper:2181 -bootstrapConfigPaths /var/pinot/config/controller.conf /var/pinot/config/broker.conf /var/pinot/config/server.conf
volumes:
- "./packages/e2e/apache-pinot:/var/pinot/config"
depends_on:
- zookeeper
# ports:
# - 9000:9000
```

broker.conf:
```
pinot.service.role=BROKER
pinot.broker.client.queryPort=8099
pinot.broker.routing.table.builder.class=random
```

controller.conf:
```
pinot.service.role=CONTROLLER
controller.port=9000
controller.vip.host=pinot-controller
controller.vip.port=9000
controller.data.dir=/var/pinot/controller/data
controller.zk.str=zookeeper:2181
```

server.conf:
```
pinot.service.role=SERVER
pinot.server.netty.port=8098
pinot.server.adminapi.port=8097
pinot.server.instance.dataDir=/var/pinot/server/data/index
pinot.server.instance.segmentTarDir=/var/pinot/server/data/segment
```

But sometimes, the broker fail to start with the following error in logs:
```
2021-03-11T14:22:02.1567273Z pinot_1 | Failed to start a Pinot [BROKER] at 24.895 since launch
2021-03-11T14:22:02.1568445Z pinot_1 | java.lang.IllegalStateException: zip file closed
2021-03-11T14:22:02.1569607Z pinot_1 | at java.util.zip.ZipFile.ensureOpen(ZipFile.java:686) ~[?:1.8.0_275]
2021-03-11T14:22:02.1570744Z pinot_1 | at java.util.zip.ZipFile.access$200(ZipFile.java:60) ~[?:1.8.0_275]
2021-03-11T14:22:02.1571931Z pinot_1 | at java.util.zip.ZipFile$ZipEntryIterator.hasNext(ZipFile.java:508) ~[?:1.8.0_275]
2021-03-11T14:22:02.1573506Z pinot_1 | at java.util.zip.ZipFile$ZipEntryIterator.hasMoreElements(ZipFile.java:503) ~[?:1.8.0_275]
2021-03-11T14:22:02.1575015Z pinot_1 | at java.util.jar.JarFile$JarEntryIterator.hasNext(JarFile.java:253) ~[?:1.8.0_275]
2021-03-11T14:22:02.1576425Z pinot_1 | at java.util.jar.JarFile$JarEntryIterator.hasMoreElements(JarFile.java:262) ~[?:1.8.0_275]
2021-03-11T14:22:02.1578434Z pinot_1 | at org.reflections.vfs.ZipDir$1$1.computeNext(ZipDir.java:30) ~[pinot-all-0.6.0-jar-with-dependencies.jar:0.6.0-bb646baceafcd9b849a1ecdec7a11203c7027e21]
2021-03-11T14:22:02.1580893Z pinot_1 | at org.reflections.vfs.ZipDir$1$1.computeNext(ZipDir.java:26) ~[pinot-all-0.6.0-jar-with-dependencies.jar:0.6.0-bb646baceafcd9b849a1ecdec7a11203c7027e21]
2021-03-11T14:22:02.1584116Z pinot_1 | at shaded.com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:145) ~[pinot-all-0.6.0-jar-with-dependencies.jar:0.6.0-bb646baceafcd9b849a1ecdec7a11203c7027e21]
2021-03-11T14:22:02.1587855Z pinot_1 | at shaded.com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:140) ~[pinot-all-0.6.0-jar-with-dependencies.jar:0.6.0-bb646baceafcd9b849a1ecdec7a11203c7027e21]
2021-03-11T14:22:02.1590879Z pinot_1 | at org.reflections.Reflections.scan(Reflections.java:243) ~[pinot-all-0.6.0-jar-with-dependencies.jar:0.6.0-bb646baceafcd9b849a1ecdec7a11203c7027e21]
2021-03-11T14:22:02.1593449Z pinot_1 | at org.reflections.Reflections.scan(Reflections.java:202) ~[pinot-all-0.6.0-jar-with-dependencies.jar:0.6.0-bb646baceafcd9b849a1ecdec7a11203c7027e21]
2021-03-11T14:22:02.1596113Z pinot_1 | at org.reflections.Reflections.(Reflections.java:123) ~[pinot-all-0.6.0-jar-with-dependencies.jar:0.6.0-bb646baceafcd9b849a1ecdec7a11203c7027e21]
2021-03-11T14:22:02.1598881Z pinot_1 | at io.swagger.jaxrs.config.BeanConfig.classes(BeanConfig.java:276) ~[pinot-all-0.6.0-jar-with-dependencies.jar:0.6.0-bb646baceafcd9b849a1ecdec7a11203c7027e21]
2021-03-11T14:22:02.1601766Z pinot_1 | at io.swagger.jaxrs.config.BeanConfig.scanAndRead(BeanConfig.java:240) ~[pinot-all-0.6.0-jar-with-dependencies.jar:0.6.0-bb646baceafcd9b849a1ecdec7a11203c7027e21]
2021-03-11T14:22:02.1604651Z pinot_1 | at io.swagger.jaxrs.config.BeanConfig.setScan(BeanConfig.java:221) ~[pinot-all-0.6.0-jar-with-dependencies.jar:0.6.0-bb646baceafcd9b849a1ecdec7a11203c7027e21]
2021-03-11T14:22:02.1608446Z pinot_1 | at org.apache.pinot.broker.broker.BrokerAdminApiApplication.setupSwagger(BrokerAdminApiApplication.java:77) ~[pinot-all-0.6.0-jar-with-dependencies.jar:0.6.0-bb646baceafcd9b849a1ecdec7a11203c7027e21]
2021-03-11T14:22:02.1612534Z pinot_1 | at org.apache.pinot.broker.broker.BrokerAdminApiApplication.start(BrokerAdminApiApplication.java:65) ~[pinot-all-0.6.0-jar-with-dependencies.jar:0.6.0-bb646baceafcd9b849a1ecdec7a11203c7027e21]
2021-03-11T14:22:02.1616505Z pinot_1 | at org.apache.pinot.broker.broker.helix.HelixBrokerStarter.start(HelixBrokerStarter.java:248) ~[pinot-all-0.6.0-jar-with-dependencies.jar:0.6.0-bb646baceafcd9b849a1ecdec7a11203c7027e21]
2021-03-11T14:22:02.1620225Z pinot_1 | at org.apache.pinot.tools.service.PinotServiceManager.startBroker(PinotServiceManager.java:135) ~[pinot-all-0.6.0-jar-with-dependencies.jar:0.6.0-bb646baceafcd9b849a1ecdec7a11203c7027e21]
2021-03-11T14:22:02.1624058Z pinot_1 | at org.apache.pinot.tools.service.PinotServiceManager.startRole(PinotServiceManager.java:93) ~[pinot-all-0.6.0-jar-with-dependencies.jar:0.6.0-bb646baceafcd9b849a1ecdec7a11203c7027e21]
2021-03-11T14:22:02.1628130Z pinot_1 | at org.apache.pinot.tools.admin.command.StartServiceManagerCommand$1.lambda$run$0(StartServiceManagerCommand.java:260) ~[pinot-all-0.6.0-jar-with-dependencies.jar:0.6.0-bb646baceafcd9b849a1ecdec7a11203c7027e21]
2021-03-11T14:22:02.1632631Z pinot_1 | at org.apache.pinot.tools.admin.command.StartServiceManagerCommand.startPinotService(StartServiceManagerCommand.java:286) [pinot-all-0.6.0-jar-with-dependencies.jar:0.6.0-bb646baceafcd9b849a1ecdec7a11203c7027e21]
2021-03-11T14:22:02.1637612Z pinot_1 | at org.apache.pinot.tools.admin.command.StartServiceManagerCommand.access$000(StartServiceManagerCommand.java:57) [pinot-all-0.6.0-jar-with-dependencies.jar:0.6.0-bb646baceafcd9b849a1ecdec7a11203c7027e21]
2021-03-11T14:22:02.1641976Z pinot_1 | at org.apache.pinot.tools.admin.command.StartServiceManagerCommand$1.run(StartServiceManagerCommand.java:260) [pinot-all-0.6.0-jar-with-dependencies.jar:0.6.0-bb646baceafcd9b849a1ecdec7a11203c7027e21]
```

Leading to an error when I try to create a table: `Failed to find instances with tag: DefaultTenant_BROKER for table...`

Contributor guide

Open the contributing guide

Research direction

Reproduce the startup failure with the supplied docker-compose configuration and the broker.conf, controller.conf, and server.conf files. Start at BrokerAdminApiApplication.setupSwagger and the org.reflections.vfs.ZipDir stack frames, then verify that the broker starts and table creation no longer reports a missing DefaultTenant_BROKER instance.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker-compose, java
Domain
databases, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.