Error: `missing required field ColumnIndex.null_pages` when loading page indexes
- Dominant language
- Rust
- Stars
- 3.6k
- Forks
- 1.3k
- Avg merge
- 2d 18h
- Merged PRs (30d)
- 169
Description
**Describe the bug**
If the `ParquetMetadataReader` tries to read metadata written by `ParquetMetaDataWriter` without first loading the page indexes, you get an error like "missing required field ColumnIndex.null_pages"
Nite this depends on https://github.com/apache/arrow-rs/pull/6463
**To Reproduce**
The full reproducer is in https://github.com/apache/arrow-rs/pull/6463. Here is the relevant piece
```rust
let parquet_bytes = create_parquet_file();
// read the metadata from the file WITHOUT the page index structures
let original_metadata = ParquetMetaDataReader::new()
.parse_and_finish(&parquet_bytes)
.unwrap();
// read metadata back from the serialized bytes requesting to read the offsets
let metadata_bytes = metadata_to_bytes(&original_metadata);
let roundtrip_metadata = ParquetMetaDataReader::new()
.with_page_indexes(true) // there are no page indexes in the metadata
.parse_and_finish(&metadata_bytes)
.unwrap(); // <******* This fails
```
**Expected behavior**
The reader should not error
I am not sure if the right fix is to
1. change the ParquetMetadataWriter to clear the index offset fields befor writing them
2. change the ParquetMetadataReader to ignore bogus offsets
3. SOmething else
**Additional context**
@etseidl has added the APIs in https://github.com/apache/arrow-rs/pull/6431
Contributor guide
Research direction
Start with the ParquetMetaDataReader and ParquetMetaDataWriter APIs involved in the reproducer from arrow-rs pull request 6463, and review the page-index APIs added in pull request 6431. Determine whether writer or reader handling should change; done means the shown metadata round trip with with_page_indexes(true) completes without the missing ColumnIndex.null_pages error.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- data-engineering
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100