apache / apache/arrow-rs

Relax column type check in RecordBatch initialization

Open
#3,605 4 comments 0 reactions 1 assignee Claimed by @viirya View on GitHub
enhancement
Dominant language
Rust
Stars
3.6k
Forks
1.3k
Avg merge
2d 18h
Merged PRs (30d)
169

Description

**Is your feature request related to a problem or challenge? Please describe what you are trying to do.**

While working on internal project incorporating Arrow/DataFusion, we encounter some issues on reading non-dictionary and dictionary arrays for same column.

During initializatoin `RecordBatch` checks if each field type in given schema matches column type in arrays. But two issues are annoying:

1. Before reading in arrays, we only know datatype of column (not arrow datatype), so we don't know if it is dictionary or not. I.e., we only know it is `T` but don't know it is `Dictionary<_, T>`.
2. While reading different files, it is possible that same column is only dictionary encoded in some files instead of all files. It causes schema mismatch later.

I'm wondering if we can relax the column type check to allow compatible one like `T` to `Dictionary<_, T>`.

**Describe the solution you'd like**

**Describe alternatives you've considered**

We could not do this.

**Additional context**

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.