apache / apache/arrow

[R] ChunkedArray$create assumes all chunks are the same type

Open
#27,116 0 comments 0 reactions 0 assignees View on GitHub
Component: R Priority: Medium Status: needs champion Type: bug
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 13h
Merged PRs (30d)
88

Description

It detects the type from the first chunk and uses it for all chunks. Normally this works ok, but it can lead to unexpected behavior, such as:

```r

data <- list(1:10, NaN)
x <- chunked_array(!!!data)
```

returns:
`Error: Invalid: Value is too large to fit in C integer type`

There are a few things that might fix/change this:
- improved error message
- chunked arrays not assuming the first chunk's types can be cast safely to all others

Note that in this case, specifying the type to int64() does "work" with an overflowed NaN value (-9223372036854775808)

```r

data <- list(1:10, NaN)
x <- chunked_array(!!!data, type = int64())
```

**Reporter**: [Jonathan Keane](https://issues.apache.org/jira/browse/ARROW-11211) / @jonkeane

**Note**: *This issue was originally created as [ARROW-11211](https://issues.apache.org/jira/browse/ARROW-11211). Please see the [migration documentation](https://github.com/apache/arrow/issues/14542) for further details.*

Contributor guide

Open the contributing guide

Research direction

Start at the R entry points ChunkedArray$create and chunked_array, then reproduce the reported list(1:10, NaN) example and compare it with the explicit int64() case. Determine the intended behavior for mixed chunk types and define completion as consistent handling with an informative result or error.

Written by the indexing model from the issue text.

Assessment

Tech stack
r
Domain
data
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
32/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.