Support more complex types when reading into arrow record batch.
Open
- Dominant language
- Rust
- Stars
- 1.4k
- Forks
- 567
- Avg merge
- 2d 2h
- Merged PRs (30d)
- 93
Description
This is a follow up issue of #244, where we have some limitations of reading into arrow record batch:
1. Only primitive type is supported. That's the projected fields count not be either nested filed of structs, or non primitive types such as map, list.
2. Type promotion is missing.
3. Default value is not supported yet.
We should implement sth like [ArrowProjectionVisitor](https://github.com/apache/iceberg-python/blob/afdfa351119090f09d38ef72857d6303e691f5ad/pyiceberg/io/pyarrow.py#L1135) to support these.
Contributor guide
Assessment
This issue has not been assessed yet.