apache / apache/arrow-rs

Parquet StatisticsConverter does not work for struct columns

Open
#7,364 0 comments 2 reactions 0 assignees View on GitHub
bug
Dominant language
Rust
Stars
3.6k
Forks
1.3k
Avg merge
2d 18h
Merged PRs (30d)
169

Description

**Describe the bug**

The `StatisticsConverter` produces all-null columns for struct-type fields.

**To Reproduce**

```rs
#[cfg(test)]
mod test_geoparquet {
use std::sync::Arc;

use arrow::array::AsArray;
use arrow::datatypes::Float32Type;
use object_store::aws::AmazonS3Builder;
use parquet::arrow::arrow_reader::ArrowReaderMetadata;
use parquet::arrow::async_reader::ParquetObjectReader;

use super::*;

#[tokio::test]
async fn test_struct_geoparquet() {
let store = Arc::new(
AmazonS3Builder::new()
.with_bucket_name("overturemaps-us-west-2")
.with_skip_signature(true)
.with_region("us-west-2")
.build()
.unwrap(),
);
let path = "release/2025-02-19.0/theme=addresses/type=address/part-00010-e084a2d7-fea9-41e5-a56f-e638a3307547-c000.zstd.parquet";
let mut object_reader = ParquetObjectReader::new(store, path.into());
let meta = ArrowReaderMetadata::load_async(&mut object_reader, Default::default())
.await
.unwrap();

let parquet_schema = meta.parquet_schema();
let column_desc = parquet_schema.column(2);

let min_bytes = meta
.metadata()
.row_group(0)
.column(2)
.statistics()
.unwrap()
.min_bytes_opt()
.unwrap();

let statistics_value_direct = f32::from_le_bytes(min_bytes.try_into().unwrap());

let converter =
StatisticsConverter::try_new("bbox", meta.schema(), meta.parquet_schema()).unwrap();
let mins = converter
.row_group_mins(meta.metadata().row_groups())
.unwrap();
let mins_struct = mins.as_struct();
let minx_bbox_minx = mins_struct.column(0).as_primitive::();

// This line fails
assert!(minx_bbox_minx.is_valid(0));
let statistics_value_via_converter = minx_bbox_minx.value(0);

assert_eq!(statistics_value_direct, statistics_value_via_converter)
}
}
```

**Expected behavior**

Some way to handle struct columns in `StatisticsConverter`.

**Additional context**

It looks like this issue has been documented in datafusion, before the `StatisticsConverter` was moved into `parquet` core.

- https://github.com/apache/datafusion/issues/10609
- https://github.com/apache/datafusion/issues/8334

It doesn't look like there was previously an issue in this repo for this.

Contributor guide

Open the contributing guide

Research direction

Start with the StatisticsConverter entry point and the test_struct_geoparquet reproducer shown in the issue. Run the test against the provided Parquet metadata and trace how struct columns produce row-group minima; done means the struct field is valid and its converted minimum matches the direct statistic.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
databases
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.