apache / apache/pinot

Reduce docker image size

Open
#8,718 3 comments 3 reactions 1 assignee Claimed by @npawar View on GitHub
Dominant language
Java
Stars
6.1k
Forks
1.5k
Avg merge
2d 3h
Merged PRs (30d)
195

Description

The current pinot docker image size is pretty large, 1GB after 0.10.0 release. This is majorly due to the shaded plugins.
```
144K plugins/pinot-batch-ingestion/pinot-batch-ingestion-standalone
144K plugins/pinot-batch-ingestion
87M plugins/pinot-environment/pinot-azure
87M plugins/pinot-environment
17M plugins/pinot-file-system/pinot-adls
14M plugins/pinot-file-system/pinot-gcs
12K plugins/pinot-file-system/pinot-hdfs
29M plugins/pinot-file-system/pinot-s3
60M plugins/pinot-file-system
3.4M plugins/pinot-input-format/pinot-avro
17M plugins/pinot-input-format/pinot-confluent-avro
360K plugins/pinot-input-format/pinot-csv
328K plugins/pinot-input-format/pinot-json
46M plugins/pinot-input-format/pinot-orc
49M plugins/pinot-input-format/pinot-parquet
2.0M plugins/pinot-input-format/pinot-protobuf
2.1M plugins/pinot-input-format/pinot-thrift
119M plugins/pinot-input-format
20M plugins/pinot-metrics/pinot-dropwizard
20M plugins/pinot-metrics/pinot-yammer
40M plugins/pinot-metrics
92K plugins/pinot-minion-tasks/pinot-minion-builtin-tasks
92K plugins/pinot-minion-tasks
16K plugins/pinot-segment-uploader/pinot-segment-uploader-default
16K plugins/pinot-segment-uploader
20K plugins/pinot-segment-writer/pinot-segment-writer-file-based
20K plugins/pinot-segment-writer
19M plugins/pinot-stream-ingestion/pinot-kafka-0.9
32M plugins/pinot-stream-ingestion/pinot-kafka-2.0
14M plugins/pinot-stream-ingestion/pinot-kinesis
60M plugins/pinot-stream-ingestion/pinot-pulsar
124M plugins/pinot-stream-ingestion
428M plugins
```

Here are a few ways to reduce the docker image size:
1. Remove unnecessary shaded jars, e.g. Hadoop ingestion jars.
2. Have the build docker script taking parameters for which plugins to be included. And we can just package the most popular shaded jars.
3. Have a script associated with `pinot-admin.sh` to take the env variable of loading plugins then download released shaded jars from the release maven Artifactory before startup.

For users:
1. For using jars not packaged, users can load jars when starting pinot. It requires an extra env variable to specify the plugins.

For plugin developers:
1. If the plugin is not packaged, users can build their customized pinot image by using the existing pinot image as the base image, build and copy the jar to the plugin directory then publish their own jars.

cc: @kishoreg

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.