apache / apache/parquet-format

[Format] Specify FIXED_SIZE_LIST Logical type

Open
#430 0 comments 0 reactions 0 assignees View on GitHub
Priority: Major Type: enhancement
Dominant language
Thrift
Stars
2.6k
Forks
508
Avg merge
2d 8h
Merged PRs (30d)
4

Description

[_Replicated from mailing list_](https://lists.apache.org/thread/xot5f3ghhtc82n1bf0wdl9zqwlrzqks3)

Arrow recently introduced [FixedShapeTensor](https://arrow.apache.org/docs/format/CanonicalExtensions.html#fixed-shape-tensor) and [VariableShapeTensor](https://arrow.apache.org/docs/format/CanonicalExtensions.html#variable-shape-tensor) canonical extension types that use FixedSizeList and StructArray(List, FixedSizeList) as storage respectfully. These are targeted at machine learning and scientific applications that deal with large datasets and would benefit from using Parquet as on disk storage.

However currently FixedSizeList is stored as List in Parquet which adds significant conversion overhead when reading and writing as [discussed here](https://github.com/apache/arrow/issues/34510). It would therefore be beneficial to introduce a FIXED_SIZE_LIST logical type to Parquet.

**Reporter**: [Rok Mihevc](https://issues.apache.org/jira/secure/ViewProfile.jspa?name=rok)
#### PRs and other links:
- [GitHub Pull Request #241](https://github.com/apache/parquet-format/pull/241)

**Note**: *This issue was originally created as [PARQUET-2474](https://issues.apache.org/jira/browse/PARQUET-2474). Please see the [migration documentation](https://issues.apache.org/jira/browse/PARQUET-2502) for further details.*

Contributor guide

Open the contributing guide

Research direction

Start by reading the linked Apache Parquet PR #241 and the original mailing-list discussion, then compare them with the issue's description of FixedSizeList storage. Done means the Parquet format specifies a FIXED_SIZE_LIST logical type that supports the stated FixedSizeList use case.

Written by the indexing model from the issue text.

Assessment

Domain
data-engineering
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.