[Feature] Introduce a new vector data type
- Dominant language
- Java
- Stars
- 3.4k
- Forks
- 1.4k
- Avg merge
- 1d 11h
- Merged PRs (30d)
- 396
Description
### Search before asking
- [x] I searched in the [issues](https://github.com/apache/paimon/issues) and found nothing similar.
### Motivation
[PIP-40](https://cwiki.apache.org/confluence/display/PAIMON/PIP-40%3A+Introduce+a+new+Vector+data+type)
### Solution
As discussed in [PIP-40](https://cwiki.apache.org/confluence/display/PAIMON/PIP-40%3A+Introduce+a+new+Vector+data+type), we propose introducing a dedicated vector data type in Paimon to better support storage and retrieval of vector data for AI workloads.
PIP-40 can be roughly split into two parts:
(1) introducing the vector data type itself;
(2) allowing users to specify the file format for vector data, to further optimize storage/access efficiency in mixed workloads.
For Part (1), a basic implementation is already available and includes:
- Introducing a new vector type. To avoid confusion with the existing term "Vector" in the codebase, the new type is named VecType.java.
- Providing a ColumnVector implementation for the vector type, with support in the paimon-arrow module, so Arrow-related file formats (e.g., Lance) can map FixedSizeList to VecType.
- Adding Flink-side compatibility: via configuration, a Flink Array can be stored as Paimon VecType.
For Part (2) (specifying the file format for vector data), work is still in progress.
- [x] Add a new DataType extension to represent the vector type.
- [x] Map vector type to arrow FixedSizeList, applying to arrow-based file format.
- [x] Provide compatibility at the Flink connector layer, enabling vector type read/write via Flink SQL.
- [x] Add relevant tests, including an end-to-end (E2E) test.
- [x] Support specifying the file format for the vector store.
Although the code is still in a draft state, it changes some basic interfaces (e.g., DataGetters), thus I'd like to discuss it early. Any comments on this @JingsongLi
### Anything else?
_No response_
### Are you willing to submit a PR?
- [x] I'm willing to submit a PR!
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with PIP-40 and the referenced VecType.java implementation, then inspect the paimon-arrow module and Flink connector compatibility described in the issue. Review the existing vector mapping, SQL read/write, and end-to-end tests; the remaining scope includes clarifying and completing file-format selection for vector data.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- data
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100