bug: broker crash at startup with `zip file closed`
- Dominant language
- Java
- Stars
- 6.1k
- Forks
- 1.5k
- Avg merge
- 1d 21h
- Merged PRs (30d)
- 189
Description
I'm trying to up a pinot cluster for my tests with the following docker-compose configuration:
```yaml
version: '3'
services:
zookeeper:
image: zookeeper:3.5.6
pinot:
image: apachepinot/pinot:0.6.0
command: StartServiceManager -clusterName e2e -port 9001 -zkAddress zookeeper:2181 -bootstrapConfigPaths /var/pinot/config/controller.conf /var/pinot/config/broker.conf /var/pinot/config/server.conf
volumes:
- "./packages/e2e/apache-pinot:/var/pinot/config"
depends_on:
- zookeeper
# ports:
# - 9000:9000
```
broker.conf:
```
pinot.service.role=BROKER
pinot.broker.client.queryPort=8099
pinot.broker.routing.table.builder.class=random
```
controller.conf:
```
pinot.service.role=CONTROLLER
controller.port=9000
controller.vip.host=pinot-controller
controller.vip.port=9000
controller.data.dir=/var/pinot/controller/data
controller.zk.str=zookeeper:2181
```
server.conf:
```
pinot.service.role=SERVER
pinot.server.netty.port=8098
pinot.server.adminapi.port=8097
pinot.server.instance.dataDir=/var/pinot/server/data/index
pinot.server.instance.segmentTarDir=/var/pinot/server/data/segment
```
But sometimes, the broker fail to start with the following error in logs:
```
2021-03-11T14:22:02.1567273Z pinot_1 | Failed to start a Pinot [BROKER] at 24.895 since launch
2021-03-11T14:22:02.1568445Z pinot_1 | java.lang.IllegalStateException: zip file closed
2021-03-11T14:22:02.1569607Z pinot_1 | at java.util.zip.ZipFile.ensureOpen(ZipFile.java:686) ~[?:1.8.0_275]
2021-03-11T14:22:02.1570744Z pinot_1 | at java.util.zip.ZipFile.access$200(ZipFile.java:60) ~[?:1.8.0_275]
2021-03-11T14:22:02.1571931Z pinot_1 | at java.util.zip.ZipFile$ZipEntryIterator.hasNext(ZipFile.java:508) ~[?:1.8.0_275]
2021-03-11T14:22:02.1573506Z pinot_1 | at java.util.zip.ZipFile$ZipEntryIterator.hasMoreElements(ZipFile.java:503) ~[?:1.8.0_275]
2021-03-11T14:22:02.1575015Z pinot_1 | at java.util.jar.JarFile$JarEntryIterator.hasNext(JarFile.java:253) ~[?:1.8.0_275]
2021-03-11T14:22:02.1576425Z pinot_1 | at java.util.jar.JarFile$JarEntryIterator.hasMoreElements(JarFile.java:262) ~[?:1.8.0_275]
2021-03-11T14:22:02.1578434Z pinot_1 | at org.reflections.vfs.ZipDir$1$1.computeNext(ZipDir.java:30) ~[pinot-all-0.6.0-jar-with-dependencies.jar:0.6.0-bb646baceafcd9b849a1ecdec7a11203c7027e21]
2021-03-11T14:22:02.1580893Z pinot_1 | at org.reflections.vfs.ZipDir$1$1.computeNext(ZipDir.java:26) ~[pinot-all-0.6.0-jar-with-dependencies.jar:0.6.0-bb646baceafcd9b849a1ecdec7a11203c7027e21]
2021-03-11T14:22:02.1584116Z pinot_1 | at shaded.com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:145) ~[pinot-all-0.6.0-jar-with-dependencies.jar:0.6.0-bb646baceafcd9b849a1ecdec7a11203c7027e21]
2021-03-11T14:22:02.1587855Z pinot_1 | at shaded.com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:140) ~[pinot-all-0.6.0-jar-with-dependencies.jar:0.6.0-bb646baceafcd9b849a1ecdec7a11203c7027e21]
2021-03-11T14:22:02.1590879Z pinot_1 | at org.reflections.Reflections.scan(Reflections.java:243) ~[pinot-all-0.6.0-jar-with-dependencies.jar:0.6.0-bb646baceafcd9b849a1ecdec7a11203c7027e21]
2021-03-11T14:22:02.1593449Z pinot_1 | at org.reflections.Reflections.scan(Reflections.java:202) ~[pinot-all-0.6.0-jar-with-dependencies.jar:0.6.0-bb646baceafcd9b849a1ecdec7a11203c7027e21]
2021-03-11T14:22:02.1596113Z pinot_1 | at org.reflections.Reflections.(Reflections.java:123) ~[pinot-all-0.6.0-jar-with-dependencies.jar:0.6.0-bb646baceafcd9b849a1ecdec7a11203c7027e21]
2021-03-11T14:22:02.1598881Z pinot_1 | at io.swagger.jaxrs.config.BeanConfig.classes(BeanConfig.java:276) ~[pinot-all-0.6.0-jar-with-dependencies.jar:0.6.0-bb646baceafcd9b849a1ecdec7a11203c7027e21]
2021-03-11T14:22:02.1601766Z pinot_1 | at io.swagger.jaxrs.config.BeanConfig.scanAndRead(BeanConfig.java:240) ~[pinot-all-0.6.0-jar-with-dependencies.jar:0.6.0-bb646baceafcd9b849a1ecdec7a11203c7027e21]
2021-03-11T14:22:02.1604651Z pinot_1 | at io.swagger.jaxrs.config.BeanConfig.setScan(BeanConfig.java:221) ~[pinot-all-0.6.0-jar-with-dependencies.jar:0.6.0-bb646baceafcd9b849a1ecdec7a11203c7027e21]
2021-03-11T14:22:02.1608446Z pinot_1 | at org.apache.pinot.broker.broker.BrokerAdminApiApplication.setupSwagger(BrokerAdminApiApplication.java:77) ~[pinot-all-0.6.0-jar-with-dependencies.jar:0.6.0-bb646baceafcd9b849a1ecdec7a11203c7027e21]
2021-03-11T14:22:02.1612534Z pinot_1 | at org.apache.pinot.broker.broker.BrokerAdminApiApplication.start(BrokerAdminApiApplication.java:65) ~[pinot-all-0.6.0-jar-with-dependencies.jar:0.6.0-bb646baceafcd9b849a1ecdec7a11203c7027e21]
2021-03-11T14:22:02.1616505Z pinot_1 | at org.apache.pinot.broker.broker.helix.HelixBrokerStarter.start(HelixBrokerStarter.java:248) ~[pinot-all-0.6.0-jar-with-dependencies.jar:0.6.0-bb646baceafcd9b849a1ecdec7a11203c7027e21]
2021-03-11T14:22:02.1620225Z pinot_1 | at org.apache.pinot.tools.service.PinotServiceManager.startBroker(PinotServiceManager.java:135) ~[pinot-all-0.6.0-jar-with-dependencies.jar:0.6.0-bb646baceafcd9b849a1ecdec7a11203c7027e21]
2021-03-11T14:22:02.1624058Z pinot_1 | at org.apache.pinot.tools.service.PinotServiceManager.startRole(PinotServiceManager.java:93) ~[pinot-all-0.6.0-jar-with-dependencies.jar:0.6.0-bb646baceafcd9b849a1ecdec7a11203c7027e21]
2021-03-11T14:22:02.1628130Z pinot_1 | at org.apache.pinot.tools.admin.command.StartServiceManagerCommand$1.lambda$run$0(StartServiceManagerCommand.java:260) ~[pinot-all-0.6.0-jar-with-dependencies.jar:0.6.0-bb646baceafcd9b849a1ecdec7a11203c7027e21]
2021-03-11T14:22:02.1632631Z pinot_1 | at org.apache.pinot.tools.admin.command.StartServiceManagerCommand.startPinotService(StartServiceManagerCommand.java:286) [pinot-all-0.6.0-jar-with-dependencies.jar:0.6.0-bb646baceafcd9b849a1ecdec7a11203c7027e21]
2021-03-11T14:22:02.1637612Z pinot_1 | at org.apache.pinot.tools.admin.command.StartServiceManagerCommand.access$000(StartServiceManagerCommand.java:57) [pinot-all-0.6.0-jar-with-dependencies.jar:0.6.0-bb646baceafcd9b849a1ecdec7a11203c7027e21]
2021-03-11T14:22:02.1641976Z pinot_1 | at org.apache.pinot.tools.admin.command.StartServiceManagerCommand$1.run(StartServiceManagerCommand.java:260) [pinot-all-0.6.0-jar-with-dependencies.jar:0.6.0-bb646baceafcd9b849a1ecdec7a11203c7027e21]
```
Leading to an error when I try to create a table: `Failed to find instances with tag: DefaultTenant_BROKER for table...`
Contributor guide
Research direction
Reproduce the startup failure with the supplied docker-compose configuration and the broker.conf, controller.conf, and server.conf files. Start at BrokerAdminApiApplication.setupSwagger and the org.reflections.vfs.ZipDir stack frames, then verify that the broker starts and table creation no longer reports a missing DefaultTenant_BROKER instance.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- docker-compose, java
- Domain
- databases, distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100