Support row group skipping for shredded variant columns
- Dominant language
- Java
- Stars
- 9.2k
- Forks
- 3.5k
- Avg merge
- 2d 11h
- Merged PRs (30d)
- 132
Description
### Feature Request / Improvement
Iceberg supports writing shredded variant columns to Parquet (PR #14297).
For a query like `WHERE variant_get(v, '$.price', 'int') > 100`, Iceberg can already skip Parquet files using manifest-level bounds, where the lower/upper bounds for $.price are stored as a Variant-encoded object on the v column's manifest entry (PR #15385).
For better performance, Iceberg should also skip row groups within a file: when $.price is shredded, the physical Parquet column `v.typed_value.price.typed_value` has standard Parquet min/max statistics. Iceberg should use those stats to skip row groups where max < 100, the same way ParquetMetricsRowGroupFilter handles regular column predicates.
The change is in parquet module, and should work with all engines.
**Scope**
- Add BoundExtract handling to ParquetMetricsRowGroupFilter
- Map variant path → physical shredded Parquet column
- Unshredded variant columns are not in scope (binary stats are not useful for path-based filtering)
- Test with Spark
### Query engine
None
### Willingness to contribute
- [x] I can contribute this improvement/feature independently
- [x] I would be willing to contribute this improvement/feature with guidance from the Iceberg community
- [ ] I cannot contribute this improvement/feature at this time
Contributor guide
Research direction
Start in the parquet module at ParquetMetricsRowGroupFilter and trace how BoundExtract handling reaches row-group statistics. Map the variant path to the shredded physical Parquet column, excluding unshredded variants, then validate the behavior with Spark as described in the issue; done means row groups with max below the predicate threshold are skipped.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, spark
- Domain
- data, databases
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 52/100