Support VECTOR columns in Flink CDC read paths
- Dominant language
- Java
- Stars
- 6.2k
- Forks
- 2.5k
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 111
Description
## Feature Description
Add support for reading VECTOR columns through Flink CDC paths.
Flink CDC may read records from data table base files or file slices to construct change records. These paths currently do not consistently preserve the Hudi VECTOR schema or apply the format-specific VECTOR decoding used by the regular Flink readers. As a result, CDC queries can fail or return incorrectly typed values when the selected schema contains VECTOR columns.
The issue applies to both supported base-file formats:
- Parquet stores VECTOR values using its physical binary encoding and must return them through Flink's ARRAY representation.
- Lance stores VECTOR values as fixed-size lists and must return them consistently through the same Flink ARRAY representation.
## Expected Behavior
- Flink CDC queries can read VECTOR columns from Parquet and Lance base files.
- VECTOR values are returned correctly for CDC records derived from base files and file slices.
- Nullable VECTOR values, column projection, field ordering, and CDC row kinds are preserved.
- Regular ARRAY columns continue to be handled as regular arrays.
- The behavior is consistent across the applicable COW and MOR CDC read paths.
## Scope
This work uses the existing Flink VECTOR configuration and Hudi VECTOR data model. It does not introduce a new VECTOR encoding or public API.
Parent tracking issue: #19053
Related Lance reader issue: #19821
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.