apache / apache/arrow-rs

Writing a BOOLEAN column with content-defined chunking panics with "RLE value encoder is not initialized"

Open
#10,929 1 comment 0 reactions 1 assignee Claimed by @M-Tesla View on GitHub
Dominant language
Rust
Stars
3.6k
Forks
1.3k
Avg merge
2d 14h
Merged PRs (30d)
167

Description

## Describe the bug

With content-defined chunking (CDC) enabled, writing a `BOOLEAN` column panics with `RLE value encoder is not initialized`.

CDC forces a data page break at the end of every chunk except the last:

```rust
// parquet/src/arrow/arrow_writer/mod.rs, ArrowColumnWriter::write_with_chunker
// Add a page break after each chunk except the last
if i + 1 < num_chunks {
match &mut self.writer {
ArrowColumnWriterImpl::Column(c) => c.add_data_page()?,
ArrowColumnWriterImpl::ByteArray(c) => c.add_data_page()?,
}
}
```

That break is unconditional. Writing the chunk can itself have flushed the page already, when the chunk's own values reach `data_page_size_limit` or `data_page_row_count_limit` exactly at the chunk boundary. The forced break then flushes a page with no buffered values.

`RleValueEncoder` builds its inner encoder lazily on the first `put`, so flushing before any value has been written panics:

```rust
// parquet/src/encodings/encoding/mod.rs
let rle_encoder = self
.encoder
.take()
.expect("RLE value encoder is not initialized");
```

A `BOOLEAN` column uses `RleValueEncoder` under `WriterVersion::PARQUET_2_0` (`fallback_encoding`), or when `Encoding::RLE` is set explicitly.

For every other encoding the same forced break does not panic, but writes a data page holding zero values.

Note that `data_page_size_limit` being smaller than `max_chunk_size` is a documented, supported configuration. From the `CdcOptions::max_chunk_size` docs:

> Note that the parquet writer has a related `data_page_size_limit` property that controls the maximum size of a parquet data page after encoding. While setting `data_page_size_limit` to a smaller value than `max_chunk_size` doesn't affect the chunking effectiveness, it results in more small parquet data pages.

**Affects:** `parquet` 59.2.0, and any release with content-defined chunking.

## To Reproduce

`Cargo.toml`:

```toml
[dependencies]
arrow = "59.2.0"
parquet = "59.2.0"
```

`src/main.rs`:

```rust
use arrow::array::{ArrayRef, BooleanArray, RecordBatch};
use parquet::arrow::ArrowWriter;
use parquet::file::properties::{CdcOptions, WriterProperties, WriterVersion};
use std::sync::Arc;

fn main() {
let values: Vec = (0..500_000).map(|i| i % 7 == 0).collect();
let col = Arc::new(BooleanArray::from(values)) as ArrayRef;
let batch = RecordBatch::try_from_iter([("flag", col)]).unwrap();

let props = WriterProperties::builder()
.set_writer_version(WriterVersion::PARQUET_2_0)
.set_data_page_size_limit(1024)
.set_content_defined_chunking(Some(CdcOptions {
min_chunk_size: 8 * 1024,
max_chunk_size: 16 * 1024,
norm_level: 0,
}))
.build();

let mut out = Vec::new();
let mut writer = ArrowWriter::try_new(&mut out, batch.schema(), Some(props)).unwrap();
writer.write(&batch).unwrap();
writer.close().unwrap();
println!("wrote {} bytes", out.len());
}
```

```console
$ cargo run --release
thread 'main' panicked at parquet-59.2.0/src/encodings/encoding/mod.rs:250:14:
RLE value encoder is not initialized
```

The zero-value pages written for other types are visible by swapping in an `Int32Array` and counting pages:

```rust
let props = WriterProperties::builder()
.set_writer_version(WriterVersion::PARQUET_2_0)
.set_dictionary_enabled(false)
.set_data_page_row_count_limit(128)
.set_content_defined_chunking(Some(CdcOptions {
min_chunk_size: 8 * 1024,
max_chunk_size: 16 * 1024,
norm_level: 0,
}))
.build();
// ... write 500_000 values of `(0..500_000).map(|i| i % 97)` into column "a", then:
// iterate the column's pages and count those with `page.num_values() == 0`
```

```console
total data pages = 611, pages with zero values = 121
```

## Expected behavior

Writing the column succeeds, and no data page holds zero values. A forced page break with nothing buffered should be a no-op.

## Additional context

The two neighbouring call sites in `parquet/src/column/writer/mod.rs` already guard this condition, so `add_data_page` is the odd one out:

- `should_add_data_page` returns `false` when `page_metrics.num_buffered_values == 0`
- `dict_fallback` and `flush_data_pages` check `page_metrics.num_buffered_values > 0` before calling `add_data_page`

---

*This issue was written by Claude (Anthropic's AI assistant) working with @adriangb. The reproduction above was executed and its output is verbatim.*

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.