Disabling statistics across multiple columns
- Dominant language
- Java
- Stars
- 9.2k
- Forks
- 3.5k
- Avg merge
- 2d 16h
- Merged PRs (30d)
- 129
Description
### Apache Iceberg version
1.10.1
### Query engine
None
### Please describe the bug π
#12771 added a write table property (`write.parquet.stats-enabled.column.`) to allow statistics to be disabled on a per-column basis. However, it appears that this only seems to work for a single column?
When adding a couple of properties to the table to disable stats across the `wire_format_message` and `json_format_message` columns, it appeared that stats were still being written to the Parquet file for the latter column. Here's some output from the Parquet CLI/DuckDB which I used to confirm this...
```
Row group 0: count: 1000 117.33 B records start: 4 total(compressed): 114.576 kB total(uncompressed):675.036 kB
--------------------------------------------------------------------------------
type encodings count avg size nulls min / max
...
wire_format_message BINARY Z _ 1000 36.82 B
json_format_message BINARY Z _ 1000 36.02 B 0 "{"eventMetadata":{"uuid":..." / "{"eventMetadata":{"uuid":..."
...
β duckdb
DuckDB v1.4.4 (Andium) 6ddac802ff
Enter ".help" for usage hints.
Connected to a transient in-memory database.
Use ".open FILENAME" to reopen on a persistent database.
D ATTACH 'warehouse' AS iceberg_catalog (
TYPE iceberg,
ENDPOINT 'http://localhost:8181',
AUTHORIZATION_TYPE 'none'
);
D SELECT * FROM iceberg_table_properties(iceberg_catalog.events.entity_events);
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ¬ββββββββββ
β key β value β
β varchar β varchar β
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββΌββββββββββ€
β write.parquet.compression-codec β zstd β
β commit.retry.total-timeout-ms β 120000 β
β commit.retry.min-wait-ms β 3000 β
β write.parquet.stats-enabled.column.wire_format_message β false β
β write.distribution-mode β hash β
β commit.retry.num-retries β 5 β
β write.parquet.stats-enabled.column.json_format_message β false β
β commit.retry.max-wait-ms β 60000 β
β owner β root β
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ΄ββββββββββ
```
~I was able to reproduce this bug in the tests by changing the string [here](https://github.com/apache/iceberg/blob/a3c538f647a113cf61dbfd7855d9c71da8c722ba/parquet/src/test/java/org/apache/iceberg/parquet/TestParquet.java#L250) to `"false"` and flipping the boolean in the assertion [here](https://github.com/apache/iceberg/blob/a3c538f647a113cf61dbfd7855d9c71da8c722ba/parquet/src/test/java/org/apache/iceberg/parquet/TestParquet.java#L265) to `false`. This resulted in the test failing as stats were still being written for the `int_field`.~
### Willingness to contribute
- [ ] I can contribute a fix for this bug independently
- [ ] I would be willing to contribute a fix for this bug with guidance from the Iceberg community
- [x] I cannot contribute a fix for this bug at this time
Contributor guide
Research direction
Start in parquet/src/test/java/org/apache/iceberg/parquet/TestParquet.java around the referenced test and inspect how the write.parquet.stats-enabled.column. properties are applied. Reproduce the case with both wire_format_message and json_format_message disabled, then confirm the test passes only when statistics are absent for each configured column.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- databases
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 65/100