apache / apache/hudi

Support VECTOR columns in Flink CDC read paths

Closed
#19,857 0 comments 0 reactions 1 assignee Claimed by @cshuo View on GitHub
area:reader engine:flink type:feature
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

## Feature Description

Add support for reading VECTOR columns through Flink CDC paths.

Flink CDC may read records from data table base files or file slices to construct change records. These paths currently do not consistently preserve the Hudi VECTOR schema or apply the format-specific VECTOR decoding used by the regular Flink readers. As a result, CDC queries can fail or return incorrectly typed values when the selected schema contains VECTOR columns.

The issue applies to both supported base-file formats:

- Parquet stores VECTOR values using its physical binary encoding and must return them through Flink's ARRAY representation.
- Lance stores VECTOR values as fixed-size lists and must return them consistently through the same Flink ARRAY representation.

## Expected Behavior

- Flink CDC queries can read VECTOR columns from Parquet and Lance base files.
- VECTOR values are returned correctly for CDC records derived from base files and file slices.
- Nullable VECTOR values, column projection, field ordering, and CDC row kinds are preserved.
- Regular ARRAY columns continue to be handled as regular arrays.
- The behavior is consistent across the applicable COW and MOR CDC read paths.

## Scope

This work uses the existing Flink VECTOR configuration and Hudi VECTOR data model. It does not introduce a new VECTOR encoding or public API.

Parent tracking issue: #19053

Related Lance reader issue: #19821

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.