apache / apache/paimon

[Feature] Introduce a new vector data type

Open
#7,011 0 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Java
Stars
3.4k
Forks
1.4k
Avg merge
1d 11h
Merged PRs (30d)
396

Description

### Search before asking

- [x] I searched in the [issues](https://github.com/apache/paimon/issues) and found nothing similar.

### Motivation

[PIP-40](https://cwiki.apache.org/confluence/display/PAIMON/PIP-40%3A+Introduce+a+new+Vector+data+type)

### Solution

As discussed in [PIP-40](https://cwiki.apache.org/confluence/display/PAIMON/PIP-40%3A+Introduce+a+new+Vector+data+type), we propose introducing a dedicated vector data type in Paimon to better support storage and retrieval of vector data for AI workloads.

PIP-40 can be roughly split into two parts:
(1) introducing the vector data type itself;
(2) allowing users to specify the file format for vector data, to further optimize storage/access efficiency in mixed workloads.

For Part (1), a basic implementation is already available and includes:
- Introducing a new vector type. To avoid confusion with the existing term "Vector" in the codebase, the new type is named VecType.java.
- Providing a ColumnVector implementation for the vector type, with support in the paimon-arrow module, so Arrow-related file formats (e.g., Lance) can map FixedSizeList to VecType.
- Adding Flink-side compatibility: via configuration, a Flink Array can be stored as Paimon VecType.

For Part (2) (specifying the file format for vector data), work is still in progress.

- [x] Add a new DataType extension to represent the vector type.
- [x] Map vector type to arrow FixedSizeList, applying to arrow-based file format.
- [x] Provide compatibility at the Flink connector layer, enabling vector type read/write via Flink SQL.
- [x] Add relevant tests, including an end-to-end (E2E) test.
- [x] Support specifying the file format for the vector store.

Although the code is still in a draft state, it changes some basic interfaces (e.g., DataGetters), thus I'd like to discuss it early. Any comments on this @JingsongLi

### Anything else?

_No response_

### Are you willing to submit a PR?

- [x] I'm willing to submit a PR!

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with PIP-40 and the referenced VecType.java implementation, then inspect the paimon-arrow module and Flink connector compatibility described in the issue. Review the existing vector mapping, SQL read/write, and end-to-end tests; the remaining scope includes clarifying and completing file-format selection for vector data.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
data
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.