apache / apache/parquet-format
Support Int8 and Int16 as basic type
- Dominant language
- Thrift
- Stars
- 2.6k
- Forks
- 508
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 4
Description
Int8 and Int16 are not supported as basic in previos version. Using 4 bytes to store int8 seems not a good idea, which means requiring more storage and read and write very slow. Besides, it is not friendly with regular computing format, such as velox, arrow, vector and so on.
With Int8 and Int16 supported, we can get less storage and better performance on reading and writing. As for forward compatible, we can use version in FileMetaData to choose how to read parquet data.
**Reporter**: [Jackey Lee](https://issues.apache.org/jira/secure/ViewProfile.jspa?name=Jackey+Lee) / @jackylee-ch
**Note**: *This issue was originally created as [PARQUET-2133](https://issues.apache.org/jira/browse/PARQUET-2133). Please see the [migration documentation](https://issues.apache.org/jira/browse/PARQUET-2502) for further details.*
Contributor guide
Research direction
Start by reviewing the Parquet format definitions around FileMetaData and the linked migration documentation. Determine how Int8 and Int16 should be represented and versioned for forward-compatible reads; done means the format can describe and read these basic types without breaking existing data.
Written by the indexing model from the issue text.
Assessment
- Domain
- data-engineering
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100