Support an experimental Parquet VECTOR repetition type for Arrow FixedSizeList
- Dominant language
- Assembly
- Stars
- 404
- Forks
- 145
- Avg merge
- 2d 4h
- Merged PRs (30d)
- 87
Description
## Problem
Arrow `FixedSizeList` is the natural type for fixed-shape data — embeddings, images, multidimensional array scientific data - where every value has exactly `N` elements and the shape is fixed and known from the schema. Today pqarrow round-trips it through Parquet as a standard 3-level `LIST`, writing per-element repetition and definition levels for a length that never varies. For wide dense vectors that is pure overhead; [on C++ we showed ~2-10x read improved performance is possible](https://gist.github.com/rok/fe4785d4a74d2e080cbad73e88cc1bef) which motivates a denser encoding.
## Proposal
Add an experimental Parquet `VECTOR` `FieldRepetitionType` that stores a fixed number of element values per row directly, without per-element rep/def levels, and map Arrow `FixedSizeList` onto it. This is the "Option B" design from the [*Fixed-size list type for Parquet* proposal](https://docs.google.com/document/d/1nf30OqK_UqxA4YTEZQszmOBEG56m9M5mp9rIYC2SUWc/edit?tab=t.0) (and the arrow-cpp prototype, rok/arrow#51).
For initial proposal this is, **leaf-only**, but we leave the door open to potentially allow non-leaf cases later:
- A `VECTOR` column is a single primitive leaf carrying `vector_length` (`vector [N]`), not a nested group.
- Only dense, non-nullable, top-level `FixedSizeList` columns with a fixed-width primitive element are encoded as `VECTOR`. Everything else (nullable value or element, zero-length, variable-width/dictionary/extension/struct/nested-list element, or a nested `FixedSizeList`) transparently falls back to the standard `LIST` encoding. Nullable, struct, and nested vectors are follow-ups.
- Opt-in on the writer via `pqarrow.WithVectorEncoding()`; reading is automatic.
Format additions (not yet in apache/parquet-format): `FieldRepetitionType.VECTOR = 3` and `SchemaElement.vector_length` (field id 11).
## Caveat
`VECTOR` is not part of apache/parquet-format yet, so this is strictly opt-in and non-portable: files written with `VECTOR` are rejected by readers that don't understand the repetition type.
## References
- [*Fixed-size list type for Parquet* design proposal](https://docs.google.com/document/d/1nf30OqK_UqxA4YTEZQszmOBEG56m9M5mp9rIYC2SUWc/edit?tab=t.0)
- [our benchmark](https://gist.github.com/rok/fe4785d4a74d2e080cbad73e88cc1bef) - showed ~2-10x performance gap
- arrow-cpp Option B prototype: rok/arrow#51
Contributor guide
Research direction
Start at the pqarrow writer path and the WithVectorEncoding() entry point, then trace FixedSizeList handling and Parquet schema encoding. Compare the proposed VECTOR restrictions and fallback rules with existing LIST behavior; done means opt-in writing, automatic reading, correct vector_length metadata, and coverage for the stated fallback cases.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- data-engineering
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100