apache / apache/iceberg

Disabling statistics across multiple columns

Open
#15,347 5 comments 0 reactions 0 assignees View on GitHub
bug good first issue
Dominant language
Java
Stars
9.2k
Forks
3.5k
Avg merge
2d 16h
Merged PRs (30d)
129

Description

### Apache Iceberg version

1.10.1

### Query engine

None

### Please describe the bug 🐞

#12771 added a write table property (`write.parquet.stats-enabled.column.`) to allow statistics to be disabled on a per-column basis. However, it appears that this only seems to work for a single column?

When adding a couple of properties to the table to disable stats across the `wire_format_message` and `json_format_message` columns, it appeared that stats were still being written to the Parquet file for the latter column. Here's some output from the Parquet CLI/DuckDB which I used to confirm this...

```
Row group 0: count: 1000 117.33 B records start: 4 total(compressed): 114.576 kB total(uncompressed):675.036 kB
--------------------------------------------------------------------------------
type encodings count avg size nulls min / max
...
wire_format_message BINARY Z _ 1000 36.82 B
json_format_message BINARY Z _ 1000 36.02 B 0 "{"eventMetadata":{"uuid":..." / "{"eventMetadata":{"uuid":..."
...

➜ duckdb
DuckDB v1.4.4 (Andium) 6ddac802ff
Enter ".help" for usage hints.
Connected to a transient in-memory database.
Use ".open FILENAME" to reopen on a persistent database.
D ATTACH 'warehouse' AS iceberg_catalog (
TYPE iceberg,
ENDPOINT 'http://localhost:8181',
AUTHORIZATION_TYPE 'none'
);
D SELECT * FROM iceberg_table_properties(iceberg_catalog.events.entity_events);
β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
β”‚ key β”‚ value β”‚
β”‚ varchar β”‚ varchar β”‚
β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€
β”‚ write.parquet.compression-codec β”‚ zstd β”‚
β”‚ commit.retry.total-timeout-ms β”‚ 120000 β”‚
β”‚ commit.retry.min-wait-ms β”‚ 3000 β”‚
β”‚ write.parquet.stats-enabled.column.wire_format_message β”‚ false β”‚
β”‚ write.distribution-mode β”‚ hash β”‚
β”‚ commit.retry.num-retries β”‚ 5 β”‚
β”‚ write.parquet.stats-enabled.column.json_format_message β”‚ false β”‚
β”‚ commit.retry.max-wait-ms β”‚ 60000 β”‚
β”‚ owner β”‚ root β”‚
β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜
```

~I was able to reproduce this bug in the tests by changing the string [here](https://github.com/apache/iceberg/blob/a3c538f647a113cf61dbfd7855d9c71da8c722ba/parquet/src/test/java/org/apache/iceberg/parquet/TestParquet.java#L250) to `"false"` and flipping the boolean in the assertion [here](https://github.com/apache/iceberg/blob/a3c538f647a113cf61dbfd7855d9c71da8c722ba/parquet/src/test/java/org/apache/iceberg/parquet/TestParquet.java#L265) to `false`. This resulted in the test failing as stats were still being written for the `int_field`.~

### Willingness to contribute

- [ ] I can contribute a fix for this bug independently
- [ ] I would be willing to contribute a fix for this bug with guidance from the Iceberg community
- [x] I cannot contribute a fix for this bug at this time

Contributor guide

Open the contributing guide

Research direction

Start in parquet/src/test/java/org/apache/iceberg/parquet/TestParquet.java around the referenced test and inspect how the write.parquet.stats-enabled.column. properties are applied. Reproduce the case with both wire_format_message and json_format_message disabled, then confirm the test passes only when statistics are absent for each configured column.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
databases
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
65/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.