apache / apache/arrow-rs

Error: `missing required field ColumnIndex.null_pages` when loading page indexes

Open
#6,464 4 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Rust
Stars
3.6k
Forks
1.3k
Avg merge
2d 18h
Merged PRs (30d)
169

Description

**Describe the bug**
If the `ParquetMetadataReader` tries to read metadata written by `ParquetMetaDataWriter` without first loading the page indexes, you get an error like "missing required field ColumnIndex.null_pages"

Nite this depends on https://github.com/apache/arrow-rs/pull/6463

**To Reproduce**
The full reproducer is in https://github.com/apache/arrow-rs/pull/6463. Here is the relevant piece

```rust
let parquet_bytes = create_parquet_file();

// read the metadata from the file WITHOUT the page index structures
let original_metadata = ParquetMetaDataReader::new()
.parse_and_finish(&parquet_bytes)
.unwrap();

// read metadata back from the serialized bytes requesting to read the offsets
let metadata_bytes = metadata_to_bytes(&original_metadata);
let roundtrip_metadata = ParquetMetaDataReader::new()
.with_page_indexes(true) // there are no page indexes in the metadata
.parse_and_finish(&metadata_bytes)
.unwrap(); // <******* This fails
```

**Expected behavior**
The reader should not error

I am not sure if the right fix is to
1. change the ParquetMetadataWriter to clear the index offset fields befor writing them
2. change the ParquetMetadataReader to ignore bogus offsets
3. SOmething else

**Additional context**
@etseidl has added the APIs in https://github.com/apache/arrow-rs/pull/6431

Contributor guide

Open the contributing guide

Research direction

Start with the ParquetMetaDataReader and ParquetMetaDataWriter APIs involved in the reproducer from arrow-rs pull request 6463, and review the page-index APIs added in pull request 6431. Determine whether writer or reader handling should change; done means the shown metadata round trip with with_page_indexes(true) completes without the missing ColumnIndex.null_pages error.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
data-engineering
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.