apache / apache/iceberg-rust

Support more complex types when reading into arrow record batch.

Open
#405 15 comments 1 reaction 1 assignee Claimed by @sdd View on GitHub
Dominant language
Rust
Stars
1.4k
Forks
567
Avg merge
2d 2h
Merged PRs (30d)
93

Description

This is a follow up issue of #244, where we have some limitations of reading into arrow record batch:

1. Only primitive type is supported. That's the projected fields count not be either nested filed of structs, or non primitive types such as map, list.
2. Type promotion is missing.
3. Default value is not supported yet.

We should implement sth like [ArrowProjectionVisitor](https://github.com/apache/iceberg-python/blob/afdfa351119090f09d38ef72857d6303e691f5ad/pyiceberg/io/pyarrow.py#L1135) to support these.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.