apache / apache/pinot

Too large tarball size

Open
#9,334 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
6.1k
Forks
1.5k
Avg merge
2d 3h
Merged PRs (30d)
195

Description

Due to the plugin structure we introduced and all the new plugins we are adding, the pinot tarball size is increasing and the size is beyond 500MB.
This has exceeded the Apache limit on the tarball size. So we should think of how to reduce it. Just throw some random thoughts:
1. Don't pack existing pinot-core contained libs into the plugin shaded jars, as they are all provided.
2. Make plugin jars downloadable/loadable from other locations, e.g. from a URI, so we can publish both nightly snapshots and release shaded jars there. But this introduces a problem for local plugin developers, so there should be some flag to turn off the downloading, but directly packaging plugins.

Contributor guide

Open the contributing guide

Research direction

Start by examining how the Pinot tarball is assembled and how plugin shaded jars package pinot-core libraries. Measure which plugins and bundled dependencies account for the size, then compare the proposed dependency exclusions with loading jars from external URIs. Done means bringing the distribution below the Apache 500MB limit while retaining a workable path for local plugin development.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
build-system, release
Issue type
Refactor
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
22/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.