Reduce docker image size
- Dominant language
- Java
- Stars
- 6.1k
- Forks
- 1.5k
- Avg merge
- 2d 3h
- Merged PRs (30d)
- 195
Description
The current pinot docker image size is pretty large, 1GB after 0.10.0 release. This is majorly due to the shaded plugins.
```
144K plugins/pinot-batch-ingestion/pinot-batch-ingestion-standalone
144K plugins/pinot-batch-ingestion
87M plugins/pinot-environment/pinot-azure
87M plugins/pinot-environment
17M plugins/pinot-file-system/pinot-adls
14M plugins/pinot-file-system/pinot-gcs
12K plugins/pinot-file-system/pinot-hdfs
29M plugins/pinot-file-system/pinot-s3
60M plugins/pinot-file-system
3.4M plugins/pinot-input-format/pinot-avro
17M plugins/pinot-input-format/pinot-confluent-avro
360K plugins/pinot-input-format/pinot-csv
328K plugins/pinot-input-format/pinot-json
46M plugins/pinot-input-format/pinot-orc
49M plugins/pinot-input-format/pinot-parquet
2.0M plugins/pinot-input-format/pinot-protobuf
2.1M plugins/pinot-input-format/pinot-thrift
119M plugins/pinot-input-format
20M plugins/pinot-metrics/pinot-dropwizard
20M plugins/pinot-metrics/pinot-yammer
40M plugins/pinot-metrics
92K plugins/pinot-minion-tasks/pinot-minion-builtin-tasks
92K plugins/pinot-minion-tasks
16K plugins/pinot-segment-uploader/pinot-segment-uploader-default
16K plugins/pinot-segment-uploader
20K plugins/pinot-segment-writer/pinot-segment-writer-file-based
20K plugins/pinot-segment-writer
19M plugins/pinot-stream-ingestion/pinot-kafka-0.9
32M plugins/pinot-stream-ingestion/pinot-kafka-2.0
14M plugins/pinot-stream-ingestion/pinot-kinesis
60M plugins/pinot-stream-ingestion/pinot-pulsar
124M plugins/pinot-stream-ingestion
428M plugins
```
Here are a few ways to reduce the docker image size:
1. Remove unnecessary shaded jars, e.g. Hadoop ingestion jars.
2. Have the build docker script taking parameters for which plugins to be included. And we can just package the most popular shaded jars.
3. Have a script associated with `pinot-admin.sh` to take the env variable of loading plugins then download released shaded jars from the release maven Artifactory before startup.
For users:
1. For using jars not packaged, users can load jars when starting pinot. It requires an extra env variable to specify the plugins.
For plugin developers:
1. If the plugin is not packaged, users can build their customized pinot image by using the existing pinot image as the base image, build and copy the jar to the plugin directory then publish their own jars.
cc: @kishoreg
Contributor guide
Assessment
This issue has not been assessed yet.