apache / apache/parquet-format

Support Int8 and Int16 as basic type

Open
#397 2 comments 0 reactions 0 assignees View on GitHub
Priority: Major Type: enhancement
Dominant language
Thrift
Stars
2.6k
Forks
508
Avg merge
2d 8h
Merged PRs (30d)
4

Description

 Int8 and Int16 are not supported as basic in previos version. Using 4 bytes to store int8 seems not a good idea, which means requiring more storage and read and write very slow. Besides, it is not friendly with regular computing format, such as velox, arrow, vector and so on.

With Int8 and Int16 supported, we can get less storage and better performance on reading and writing. As for forward compatible, we can use version in FileMetaData to choose how to read parquet data.

**Reporter**: [Jackey Lee](https://issues.apache.org/jira/secure/ViewProfile.jspa?name=Jackey+Lee) / @jackylee-ch

**Note**: *This issue was originally created as [PARQUET-2133](https://issues.apache.org/jira/browse/PARQUET-2133). Please see the [migration documentation](https://issues.apache.org/jira/browse/PARQUET-2502) for further details.*

Contributor guide

Open the contributing guide

Research direction

Start by reviewing the Parquet format definitions around FileMetaData and the linked migration documentation. Determine how Int8 and Int16 should be represented and versioned for forward-compatible reads; done means the format can describe and read these basic types without breaking existing data.

Written by the indexing model from the issue text.

Assessment

Domain
data-engineering
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.