Remove raw data from binary release
- Dominant language
- Java
- Stars
- 6.1k
- Forks
- 1.5k
- Avg merge
- 1d 21h
- Merged PRs (30d)
- 189
Description
Currently, our binary distribution packs ~100MB raw data for pinot quick starter scripts. Removing this can greatly reduce the size of our official binary distribution, which is currently over 500MB.
```
7.5M ./examples/minions/batch/baseballStats/rawdata/baseballStats_data.csv
21M ./examples/batch/githubEvents/rawdata_json_index/githubEvents_data.json
20M ./examples/batch/githubEvents/rawdata_complexTypeHandling/githubEvents_data.json
7.5M ./examples/batch/baseballStats/rawdata/baseballStats_data.csv
31M ./examples/stream/airlineStats/sample_data/airlineStats_data.json
3.0M ./examples/stream/airlineStats/sample_data/airlineStats_data.avro
...
```
For the above files, we should change the script to download instead of packing along with the release.
Contributor guide
Research direction
Start by locating the release or quick starter scripts responsible for packaging the raw data listed under examples/minions, examples/batch, and examples/stream. Check how the binary archive is assembled, then verify the scripts download those files at runtime and the resulting distribution no longer includes them.
Written by the indexing model from the issue text.
Assessment
- Domain
- release
- Issue type
- Feature
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100