apache / apache/parquet-format

How to configure/tweak Parquet for storing and retrieving vector embeddings?

Open
#553 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Thrift
Stars
2.6k
Forks
508
Avg merge
2d 8h
Merged PRs (30d)
4

Description

### Describe the usage question you have. Please include as many useful details as possible.

Hi Parquet community,

Im an engineer that works in open source data infra primarily in the java ecosystem. I wanted to ask the community some questions for how to better configure/tune parquet today for storing and retrieval of vectors. Also interested in the future work around this item as well but for now curious on what users can tweak.

Typically from what i've seen most models will generate vector embeddings as an array of floating point values, with dimensions around 700-1500 elements (taking up around 3KB - 6KB per vector), so my questions will be based on this input.
* https://developers.openai.com/api/docs/guides/embeddings
* https://docs.cloud.google.com/vertex-ai/generative-ai/docs/embeddings

1. Since there is currently no native logical `VECTOR` type within Parquet, from the existing types what is the recommended data to use type for writing vectors(assumption is most users today would try parquet's `LIST` with `FLOAT` but is there others way to represent this better)? Is there plans for Parquet to add a type such as this in the future?
2. Since vectors are high cardinality encodings such as `DICTIONARY, RLE` might not be as useful so is there a recommended encoding to leverage?
3. Is there a recommended compression codec to use for vectors? ~~If not, is there a way to disable compression per column within parquet java~~?
4. Should users be disabling stats on these columns?
5. Is there a recommendation for tuning row group and page size for vectors?

Thanks again for your assistance and help, if there is any roadmap specifically you can point me to would be highly appreciated in terms of changes happening in parquet around this.

cc @julienledem @emkornfield

Contributor guide

Open the contributing guide

Research direction

No source file or test is identified in the issue. Start by reviewing Parquet's existing LIST/FLOAT representation and the Parquet Java configuration points related to encoding, compression, statistics, row groups, and page size. Done would be documented, version-specific guidance addressing the five questions and any relevant roadmap references.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
data-engineering
Issue type
Documentation
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.