apache / apache/pinot

Remove raw data from binary release

Open
#7,240 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
6.1k
Forks
1.5k
Avg merge
1d 21h
Merged PRs (30d)
189

Description

Currently, our binary distribution packs ~100MB raw data for pinot quick starter scripts. Removing this can greatly reduce the size of our official binary distribution, which is currently over 500MB.

```
7.5M ./examples/minions/batch/baseballStats/rawdata/baseballStats_data.csv
21M ./examples/batch/githubEvents/rawdata_json_index/githubEvents_data.json
20M ./examples/batch/githubEvents/rawdata_complexTypeHandling/githubEvents_data.json
7.5M ./examples/batch/baseballStats/rawdata/baseballStats_data.csv
31M ./examples/stream/airlineStats/sample_data/airlineStats_data.json
3.0M ./examples/stream/airlineStats/sample_data/airlineStats_data.avro
...
```

For the above files, we should change the script to download instead of packing along with the release.

Contributor guide

Open the contributing guide

Research direction

Start by locating the release or quick starter scripts responsible for packaging the raw data listed under examples/minions, examples/batch, and examples/stream. Check how the binary archive is assembled, then verify the scripts download those files at runtime and the resulting distribution no longer includes them.

Written by the indexing model from the issue text.

Assessment

Domain
release
Issue type
Feature
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.