apache / apache/arrow-rs

Support nested list casting

Open
#4,483 0 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Rust
Stars
3.6k
Forks
1.3k
Avg merge
2d 18h
Merged PRs (30d)
169

Description

**Is your feature request related to a problem or challenge? Please describe what you are trying to do.**

Currently, the implementation of typecasting in the ListArray assumes that the inner elements have the same dimension as the targeted ListArray. However, there is a need to support nested list casting, where the internal elements can have a different dimension than the targeted ListArray.

For example,
Array: ListArray[PrimitiveArray[1,2,3]]
Targeted Type: List(List(Int32))
Current result: ListArray[ListArray[PrimitiveArray[1], PrimitiveArray[2], PrimitiveArray[3]]]
Expected result: ListArray[ListArray[PrimitiveArray[1,2,3]]

**Describe the solution you'd like**

Casting List to the same dimensions first, then cast the inner primitive type. *Only casting from lower dimension to high dimension is allowed*.

Additional: Adding `fn get_list_dimension` for DataType::List is also helpful.

**Describe alternatives you've considered**

**Additional context**

This feature is helpful for https://github.com/apache/arrow-datafusion/issues/6849

Self-assigned

Contributor guide

Open the contributing guide

Research direction

Start by locating the ListArray typecasting implementation and the DataType::List definition mentioned in the issue; inspect how dimensions are currently handled. Add support for casting lower-dimensional lists to higher-dimensional lists before casting the inner primitive type, and consider the requested get_list_dimension helper. Done means the provided ListArray example produces the expected nested result and higher-dimensional casting is covered.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
data-engineering
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
40/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.