apache / apache/parquet-format
support saving meta and data seperately
- Dominant language
- Thrift
- Stars
- 2.6k
- Forks
- 508
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 4
Description
I often needs to create tens of milliions of small dataframes and save them into parquet files. all these dataframes have the same column and index information. and normally they have the same number of rows(around 300).
as the data is quite small, the parquet meta information is relatively large and it's quite a big waste of disk space, as the same meta information is repeated tens of millions of times.
concating them into one big parquet file can save disk space, but it's not friendly for parallel processing of each small dataframe.
if I can save one copy of the meta information into one file, and the rest parquet files contains only the data. then the disk space can be saved, and still good for parallel processing.
seems to me this is possible by design, but I couldn't find any API supporting this.
**Reporter**: [lei yu](https://issues.apache.org/jira/secure/ViewProfile.jspa?name=assassin5615)
**Note**: *This issue was originally created as [PARQUET-2207](https://issues.apache.org/jira/browse/PARQUET-2207). Please see the [migration documentation](https://issues.apache.org/jira/browse/PARQUET-2502) for further details.*
Contributor guide
Research direction
The issue names no files, tests, or entry points. Start by reviewing the Parquet format specification and the repository's existing metadata and data-layout definitions, then determine whether separate shared metadata and data-only files can be represented compatibly. Done means the format and API behavior are specified clearly enough for implementations and parallel readers to rely on them.
Written by the indexing model from the issue text.
Assessment
- Domain
- data-engineering
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100