Investigate how we can reduce the size of binary distribution
- Dominant language
- Java
- Stars
- 14.1k
- Forks
- 3.8k
- Avg merge
- 2d 58m
- Merged PRs (30d)
- 233
Description
There is a limit on the max file size that can be committed to the asf subversion server. The max size is currently set to 450MB for us. The binary size of 0.21.0 is 358MB. Looking more closely at what are occupying much space, the core extensions contribute more than half of the total binary size.
```
➜ apache-druid-0.21.0 git:(733697c25f) du -d 1 . | sort -rn
407196 .
259932 ./extensions
111668 ./lib
31132 ./hadoop-dependencies
2488 ./quickstart
860 ./licenses
808 ./conf
148 ./bin
```
Here is the size of core extensions.
```
➜ extensions git:(733697c25f) du -d 1 . | sort -rn
259932 .
63412 ./druid-ranger-security
49192 ./druid-kubernetes-extensions
41260 ./druid-parquet-extensions
28168 ./druid-hdfs-storage
15616 ./druid-orc-extensions
11160 ./druid-kafka-extraction-namespace
9796 ./druid-kafka-indexing-service
6760 ./druid-avro-extensions
6428 ./druid-aws-rds-extensions
5288 ./druid-kerberos
3856 ./druid-azure-extensions
3848 ./druid-pac4j
2956 ./druid-kinesis-indexing-service
2396 ./druid-datasketches
2228 ./druid-stats
2000 ./druid-protobuf-extensions
1984 ./druid-lookups-cached-single
1408 ./druid-lookups-cached-global
936 ./postgresql-metadata-storage
576 ./druid-bloom-filter
228 ./druid-google-extensions
160 ./druid-basic-security
116 ./druid-histogram
80 ./druid-s3-extensions
36 ./druid-ec2-extensions
24 ./mysql-metadata-storage
16 ./simple-client-sslcontext
```
As we are going to keep adding new extensions in the future, 100MB might not be a lot of space left. We should investigate how we can reduce the size of these extensions.
Contributor guide
Research direction
Start by examining the extensions directory and the binary distribution produced for version 0.21.0. Re-run the reported du measurements to identify the largest extension contributions and investigate how those artifacts are included in the distribution. Done means documenting a concrete, validated way to reduce the distribution size.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- build-system, release
- Issue type
- Refactor
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100