Support reading and writing VECTOR columns for Flink with the Lance file format
- Dominant language
- Java
- Stars
- 6.2k
- Forks
- 2.5k
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 111
Description
---
## Feature Description
**What the feature achieves:**
Add VECTOR column read/write support to the Flink **Lance** base-file path.
Hudi already supports VECTOR columns end-to-end on **Spark + Lance** and on **Flink + Parquet**. However, **Flink + Lance** currently supports neither writing nor reading VECTOR columns (in fact, not even plain `ARRAY` columns).
This feature implements the missing encode/decode so that a VECTOR column can be written to and read from a Flink Lance base file, and—critically—keeps the on-disk encoding **strictly aligned with the Spark Lance path** so files are cross-engine readable.
**Why this feature is needed:**
- Vector-search / embedding workloads increasingly land through Flink streaming pipelines; Lance is the columnar format optimized for vector data, but Flink users currently cannot use VECTOR columns with Lance.
- It closes a parity gap: the same table written by Spark+Lance (or Flink+Parquet) cannot be written/read by Flink+Lance today.
- Cross-engine interoperability: a Lance file with a VECTOR column written by Flink should be readable by Spark and vice-versa.
## User Experience
**How users will use this feature:**
- **DDL / config:** declare the vector column as `ARRAY` or `ARRAY`, mark it via the existing vector-columns option (`FlinkOptions.VECTOR_COLUMNS`, e.g. `embedding:128,features:64`), and select the Lance base file format:
```sql
CREATE TABLE t (
id STRING,
embedding ARRAY, -- VECTOR(128)
ts BIGINT,
PRIMARY KEY (id) NOT ENFORCED
) WITH (
'connector' = 'hudi',
'path' = '...',
'vector.columns' = 'embedding:128',
'hoodie.table.base.file.format' = 'LANCE'
);
```
- **Behavior:** insert / read / column projection / MOR upsert behave the same as on the Parquet base file format (matching the existing `ITTestVectorDataSource` coverage).
- **Element types:** `FLOAT` and `DOUBLE` are supported. `INT8` (`ARRAY` vector) is **rejected with a clear error**, matching the current Spark Lance writer limitation (the lance-spark connector only treats FLOAT/DOUBLE as fixed-size-list, and the Spark reader only restores FLOAT/DOUBLE). INT8 over Lance is a coordinated cross-engine follow-up.
- **API changes:** none. Existing options are reused.
**On-disk encoding (aligned with Spark Lance):**
- A `VECTOR(dim, FLOAT|DOUBLE)` column is stored as an Arrow `FixedSizeList` with a non-nullable child.
- The Lance file footer carries `hoodie.vector.columns` (`HoodieSchema.VECTOR_COLUMNS_METADATA_KEY`), identical to the Spark writer, so any engine can identify and reconstruct VECTOR columns.
## Hudi RFC Requirements
**RFC PR link:** N/A
**Why RFC isn't needed:**
- Does this change public interfaces/APIs? **No** — reuses existing `vector.columns` option and the existing VECTOR data model.
- Does this change storage format? **No new format** — it reuses the exact Lance `FixedSizeList` encoding and footer metadata already produced by the Spark Lance writer; the change only adds the Flink-side encode/decode to match.
- Justification: scoped, additive engine parity with an already-shipped on-disk contract; no new public surface or format.
## Implementation notes (for reference)
- Write: thread the vector-column map (`HoodieVectorUtils.detectVectorColumns`) from `HoodieRowDataFileWriterFactory.newLanceFileWriter` into `HoodieRowDataLanceWriter`; emit `FixedSizeList` fields in a vector-aware `toArrowSchema`; write the footer via the existing `HoodieBaseLanceWriter.additionalSchemaMetadata()` hook.
- Read: use the already-passed `requestedSchema` in `HoodieRowDataLanceReader.getRowDataIterator` to detect vectors; decode `FixedSizeListVector` → `ArrayData`; map `FixedSizeList` → `ARRAY` in `toLogicalType`; restore VECTOR metadata from the footer in `getSchema()`.
- Reuse: `HoodieSchema.Vector` model + footer helpers, `HoodieVectorUtils.detectVectorColumns`, Arrow `FixedSizeListVector` / `Float4Vector` / `Float8Vector`.
## Out of scope
- INT8 vectors on Lance (blocked by lance-spark; revisit cross-engine).
- Schema evolution on Flink Lance (already explicitly unsupported).
- Nested VECTOR (inside struct/array/map) — stays rejected, consistent with Spark.
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.