apache / apache/arrow-rs

Consistent Schema Enforcement

Open
#4,801 6 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Rust
Stars
3.6k
Forks
1.3k
Avg merge
2d 18h
Merged PRs (30d)
169

Description

**Is your feature request related to a problem or challenge? Please describe what you are trying to do.**

Comparing primitive arrays for equality, perhaps in the context of a compute kernel, is relatively straightforward. A `DataType::Int8` is equal to `DataType::Int8` and not equal to `DataType::UInt8`.

For nested types such as `StructArray`, `ListArray` and `RecordBatch` this gets more complex, how strictly should we enforce that a schema is consistent. Should we allow an array to be of a different type to its schema, what about nullability or metadata?

We currently have a range of approaches:

* RowConvert accepts any arrays so long as the array's `DataType::equals_datatype`, ignoring metadata and field names, but validating nullability
* Parquet's ArrowWriter accept any RecordBatch so long as the writer's `Schema::contains` the provided batch schema, this forces nullability and metadata to be a subset
* arrow-ipc, arrow-json, arrow-csv writers ignore the schema entirely, potentially allowing it to change between batches
* Most kernels and array constructors use PartialEq to enforce consistency of their inputs
* Array constructors enforce field nullability matches the presence of null buffers, this is non-trivial in the case of StructArray
* MutableArrayData uses the types of the first batch and will panic if it encounters inconsistency

**Describe the solution you'd like**

I don't really know, eagerly performing validation can help to catch bugs and issues, but on the flip side it is frustrating to be validating things like field names, metadata, or even nullability, that in most cases won't make a different to correctness

**Describe alternatives you've considered**

**Additional context**

#1888
#3226
#4799

Contributor guide

Open the contributing guide

Research direction

Start by comparing the validation behavior described for RowConvert, Parquet's ArrowWriter, arrow-ipc, arrow-json, arrow-csv, array constructors, kernels, and MutableArrayData. Review issues #1888, #3226, and #4799 for existing context. Done would require an agreed schema-consistency policy and corresponding changes across the affected components, but the issue does not define that policy.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
data
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.