apache / apache/arrow-rs

Providing schema hint to ArrowReaderMetadata causes resulting schema to omit virtual fields

Open
#11,046 0 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Rust
Stars
3.6k
Forks
1.3k
Avg merge
2d 18h
Merged PRs (30d)
169

Description

### Describe the bug

I recently had a use case where I had to supply both a schema hint and virtual columns (specifically row numbers) to `ArrowReaderOptions`, which was then used to construct an `ArrowReaderMetadata`.

The original code only asked for the virtual columns. When I changed the code to supply both virtual columns AND a schema hint, I observed some interesting behavior.

When only supplying virtual columns, the result of `ArrowReaderMetadata::schema` included the virtual column fields as supplied in `ArrowReaderOptions::with_virtual_columns`.

When supplying BOTH virtual columns and schema hint, the result of `ArrowReaderMetadata::schema` did NOT include the virtual column fields.

This caused issues for my downstream code that did some inspection on this schema to fail, since the column no longer exists.

Initially, I tried to work around this issue by including the virtual column IN the schema hint. This caused issues in `ArrowReaderMetadata::with_supplied_schema` since the number of fields in the schema did not match the number of fields in the Parquet file, with the error "Incompatible supplied Arrow schema: expected X columns received Y".

I eventually landed on a workaround specific to my use case that is not translatable to a generic solution, but this behavior still left a lot to be desired.

### To Reproduce

```rust
let row_number_field = Field::new("row_number", DataType::Int64, false).with_extension_type(RowNumber)

let options = ArrowReaderOptions::new().with_virtual_fields(vec![row_number_field]);
// uncomment this line:
// let options = options.with_schema(some_schema)

let metadata = ArrowReaderMetadata::load(some_reader, options)?;
assert!(metadata.schema().fields.find("row_number").is_some())
```

### Expected behavior

I'd expect one of two things to happen:
1. Include the virtual column in the resulting schema
2. Don't error when schema hint contains virtual columns

### Additional context

I'd understand if this behavior is not considered a bug. After all, the library is simply executing the contract of `with_schema`, but it still led to some head scratching.

I appreciate the work done for this library, thanks all!

Contributor guide

Open the contributing guide

Research direction

Start at ArrowReaderMetadata::load and ArrowReaderMetadata::with_supplied_schema, comparing how ArrowReaderOptions::with_virtual_fields and with_schema are handled. Reproduce the provided row_number example; done means the resulting schema retains virtual fields or schema hints containing them no longer trigger the field-count error.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
data
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
65/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.