Providing schema hint to ArrowReaderMetadata causes resulting schema to omit virtual fields
- Dominant language
- Rust
- Stars
- 3.6k
- Forks
- 1.3k
- Avg merge
- 2d 18h
- Merged PRs (30d)
- 169
Description
### Describe the bug
I recently had a use case where I had to supply both a schema hint and virtual columns (specifically row numbers) to `ArrowReaderOptions`, which was then used to construct an `ArrowReaderMetadata`.
The original code only asked for the virtual columns. When I changed the code to supply both virtual columns AND a schema hint, I observed some interesting behavior.
When only supplying virtual columns, the result of `ArrowReaderMetadata::schema` included the virtual column fields as supplied in `ArrowReaderOptions::with_virtual_columns`.
When supplying BOTH virtual columns and schema hint, the result of `ArrowReaderMetadata::schema` did NOT include the virtual column fields.
This caused issues for my downstream code that did some inspection on this schema to fail, since the column no longer exists.
Initially, I tried to work around this issue by including the virtual column IN the schema hint. This caused issues in `ArrowReaderMetadata::with_supplied_schema` since the number of fields in the schema did not match the number of fields in the Parquet file, with the error "Incompatible supplied Arrow schema: expected X columns received Y".
I eventually landed on a workaround specific to my use case that is not translatable to a generic solution, but this behavior still left a lot to be desired.
### To Reproduce
```rust
let row_number_field = Field::new("row_number", DataType::Int64, false).with_extension_type(RowNumber)
let options = ArrowReaderOptions::new().with_virtual_fields(vec![row_number_field]);
// uncomment this line:
// let options = options.with_schema(some_schema)
let metadata = ArrowReaderMetadata::load(some_reader, options)?;
assert!(metadata.schema().fields.find("row_number").is_some())
```
### Expected behavior
I'd expect one of two things to happen:
1. Include the virtual column in the resulting schema
2. Don't error when schema hint contains virtual columns
### Additional context
I'd understand if this behavior is not considered a bug. After all, the library is simply executing the contract of `with_schema`, but it still led to some head scratching.
I appreciate the work done for this library, thanks all!
Contributor guide
Research direction
Start at ArrowReaderMetadata::load and ArrowReaderMetadata::with_supplied_schema, comparing how ArrowReaderOptions::with_virtual_fields and with_schema are handled. Reproduce the provided row_number example; done means the resulting schema retains virtual fields or schema hints containing them no longer trigger the field-count error.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- data
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 65/100