apache / apache/iceberg

Support row group skipping for shredded variant columns

Open
#15,510 3 comments 0 reactions 0 assignees View on GitHub
improvement
Dominant language
Java
Stars
9.2k
Forks
3.5k
Avg merge
2d 11h
Merged PRs (30d)
132

Description

### Feature Request / Improvement

Iceberg supports writing shredded variant columns to Parquet (PR #14297).

For a query like `WHERE variant_get(v, '$.price', 'int') > 100`, Iceberg can already skip Parquet files using manifest-level bounds, where the lower/upper bounds for $.price are stored as a Variant-encoded object on the v column's manifest entry (PR #15385).

For better performance, Iceberg should also skip row groups within a file: when $.price is shredded, the physical Parquet column `v.typed_value.price.typed_value` has standard Parquet min/max statistics. Iceberg should use those stats to skip row groups where max < 100, the same way ParquetMetricsRowGroupFilter handles regular column predicates.

The change is in parquet module, and should work with all engines.

**Scope**
- Add BoundExtract handling to ParquetMetricsRowGroupFilter
- Map variant path → physical shredded Parquet column
- Unshredded variant columns are not in scope (binary stats are not useful for path-based filtering)
- Test with Spark

### Query engine

None

### Willingness to contribute

- [x] I can contribute this improvement/feature independently
- [x] I would be willing to contribute this improvement/feature with guidance from the Iceberg community
- [ ] I cannot contribute this improvement/feature at this time

Contributor guide

Open the contributing guide

Research direction

Start in the parquet module at ParquetMetricsRowGroupFilter and trace how BoundExtract handling reaches row-group statistics. Map the variant path to the shredded physical Parquet column, excluding unshredded variants, then validate the behavior with Spark as described in the issue; done means row groups with max below the predicate threshold are skipped.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, spark
Domain
data, databases
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
52/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.