Convert multi dimensional numpy array to pyarrow array
- Dominant language
- C++
- Stars
- 17.1k
- Forks
- 4.3k
- Avg merge
- 3d 13h
- Merged PRs (30d)
- 88
Description
`data = np.zeros((10,8), dtype=np.uint8)`
`out = pa.array(list(data))`
`out.type # ListType(list)`
`data = np.zeros((3,4,6), dtype=np.uint8)`
`out = pa.array(list(data)) # Throws error ArrowInvalid: Can only convert 1-dimensional array values`
Even though it's working on 2D numpy arrays perfectly, it doesn't work on N-Dimensional numpy arrays (where N > 2). Is possible to extend the current feature for inner elements with dimension greater than 1?
**Reporter**: [Bhavitvya Malik](https://issues.apache.org/jira/browse/ARROW-11600)
**Note**: *This issue was originally created as [ARROW-11600](https://issues.apache.org/jira/browse/ARROW-11600). Please see the [migration documentation](https://github.com/apache/arrow/issues/14542) for further details.*
Contributor guide
Research direction
Reproduce the reported pa.array(list(data)) behavior with the 2D and 3D NumPy examples from the issue. Trace the existing NumPy-to-Arrow conversion entry point and define done as successful conversion of N-dimensional inner arrays without the reported ArrowInvalid error, while preserving the existing 2D result.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- numpy, python
- Domain
- data
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100