Projection Pushdown with Newly Added Columns Fails for Old Batches
- Dominant language
- Java
- Stars
- 2.1k
- Forks
- 625
- Avg merge
- 3d 14h
- Merged PRs (30d)
- 97
Description
### Search before asking
- [x] I searched in the [issues](https://github.com/apache/fluss/issues) and found nothing similar.
### Fluss version
0.9.0 (latest release)
### Please describe the bug 🐞
Our system supports schema evolution: when a new column is added to the table schema, existing data files are not rewritten—instead, the query engine handles missing columns by injecting NULL or default values at read time (client-side compatibility). This works correctly in most scenarios.
However, when projection pushdown is applied and the query explicitly selects the newly added column, the scan operator may attempt to read the column directly from storage—despite its absence in old data files—leading to a failure because the column vector is missing in the batch.
### Solution
_No response_
### Are you willing to submit a PR?
- [ ] I'm willing to submit a PR!
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by reproducing projection pushdown against old data files after adding a column, then inspect the scan operator's handling of missing column vectors and the client-side schema-compatibility path. Done means selecting the newly added column succeeds for old batches with NULL or default values, while existing projections continue to work.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- data-engineering, databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100