apache / apache/fluss

Projection Pushdown with Newly Added Columns Fails for Old Batches

Open
#3,423 4 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
2.1k
Forks
625
Avg merge
3d 14h
Merged PRs (30d)
97

Description

### Search before asking

- [x] I searched in the [issues](https://github.com/apache/fluss/issues) and found nothing similar.

### Fluss version

0.9.0 (latest release)

### Please describe the bug 🐞

Our system supports schema evolution: when a new column is added to the table schema, existing data files are not rewritten—instead, the query engine handles missing columns by injecting NULL or default values at read time (client-side compatibility). This works correctly in most scenarios.

However, when projection pushdown is applied and the query explicitly selects the newly added column, the scan operator may attempt to read the column directly from storage—despite its absence in old data files—leading to a failure because the column vector is missing in the batch.

### Solution

_No response_

### Are you willing to submit a PR?

- [ ] I'm willing to submit a PR!

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reproducing projection pushdown against old data files after adding a column, then inspect the scan operator's handling of missing column vectors and the client-side schema-compatibility path. Done means selecting the newly added column succeeds for old batches with NULL or default values, while existing projections continue to work.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
data-engineering, databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.