[parquet] `set_dictionary_enabled(true)` is silently ignored for FIXED_LEN_BYTE_ARRAY columns at the default writer version
- Dominant language
- Rust
- Stars
- 3.6k
- Forks
- 1.3k
- Avg merge
- 2d 18h
- Merged PRs (30d)
- 169
Description
### Describe the bug
`WriterProperties::set_dictionary_enabled(true)` and `set_column_dictionary_enabled(col, true)` are both silently discarded for `FIXED_LEN_BYTE_ARRAY` columns when the writer version is `PARQUET_1_0`.
https://github.com/apache/arrow-rs/blob/afab753e84cf14fcdbe61518f3297ac38596b361/parquet/src/column/writer/encoder.rs#L237-L238
https://github.com/apache/arrow-rs/blob/afab753e84cf14fcdbe61518f3297ac38596b361/parquet/src/column/writer/mod.rs#L1811-L1821
Please note that I am not asking to change the default; it conforms with Java and f. ex. Trinos' own writer also implements the same behaviour.
I found one outlier (parquet-cpp), but that is besides the point. The only conclusion to draw from this is that readers can read those files; I created a test table with pyiceberg and read it back with iceberg-rust and data is intact, no errors.
I am just suggesting to either honor the Clients' `set_dictionary_enabled` request or "fail" at setting the value in a more obvious way.
### To Reproduce
Two columns, same file, 100k rows, 1000 distinct values each, no compression. `dictionary_enabled` set to `true` both globally and per-column.
```
let props = WriterProperties::builder()
.set_dictionary_enabled(true)
.set_column_dictionary_enabled("d".into(), true)
.set_writer_version(version)
.build();
```
```
PARQUET_1_0 (default)
col d FIXED_LEN_BYTE_ARRAY dict_page=NONE encodings=[PLAIN, RLE]
col i INT64 dict_page=present encodings=[PLAIN, RLE, RLE_DICTIONARY]
file: 1,734,680 bytes
```
```
PARQUET_2_0
col d FIXED_LEN_BYTE_ARRAY dict_page=present encodings=[PLAIN, RLE, RLE_DICTIONARY]
col i INT64 dict_page=present encodings=[PLAIN, RLE, RLE_DICTIONARY]
file: 275,932 bytes
```
### Expected behavior
Either the explicit setting is honoured, or it is rejected with an error/warning.
### Additional context
Real-world impact is a size/time trade: on a 200-column test Iceberg table with 160 decimal(38,9) columns (values from a 1000-value pool per column, zstd level 1), output went 388 → 251 bytes/row and 2.13 → 1.38 GB once dictionary encoding was reachable, at a cost of +32 s on a ~200 s run, measured twice. The point is that today the caller cannot make that trade at all.
Contributor guide
Research direction
Start with parquet/src/column/writer/encoder.rs around the linked lines and parquet/src/column/writer/mod.rs around the referenced writer-property handling. Reproduce the PARQUET_1_0 and PARQUET_2_0 configurations from the issue, then inspect the existing writer tests for a suitable assertion. Done means the explicit dictionary setting is honored for FIXED_LEN_BYTE_ARRAY columns or is rejected with an obvious error or warning.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- data-engineering
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 58/100