Relax column type check in RecordBatch initialization
- Dominant language
- Rust
- Stars
- 3.6k
- Forks
- 1.3k
- Avg merge
- 2d 18h
- Merged PRs (30d)
- 169
Description
**Is your feature request related to a problem or challenge? Please describe what you are trying to do.**
While working on internal project incorporating Arrow/DataFusion, we encounter some issues on reading non-dictionary and dictionary arrays for same column.
During initializatoin `RecordBatch` checks if each field type in given schema matches column type in arrays. But two issues are annoying:
1. Before reading in arrays, we only know datatype of column (not arrow datatype), so we don't know if it is dictionary or not. I.e., we only know it is `T` but don't know it is `Dictionary<_, T>`.
2. While reading different files, it is possible that same column is only dictionary encoded in some files instead of all files. It causes schema mismatch later.
I'm wondering if we can relax the column type check to allow compatible one like `T` to `Dictionary<_, T>`.
**Describe the solution you'd like**
**Describe alternatives you've considered**
We could not do this.
**Additional context**
Contributor guide
Assessment
This issue has not been assessed yet.