apache / apache/arrow

[R] Named lists cannot be serialized to a map column

Open
#34,409 3 comments 0 reactions 0 assignees View on GitHub
Component: R Priority: Medium Status: needs champion Type: bug
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 18h
Merged PRs (30d)
91

Description

### Describe the bug, including details regarding any error messages, version, and platform.

Let's start with a simple table that contains a named list in each row:
```R
> x = tibble::tibble(id=1:3, metadata=list(list(a=1), list(b=2, c=3), list(d=3, e=4)))
> x
# A tibble: 3 × 2
id metadata

1 1
2 2
3 3
```

From the outset, I would hope that the `metadata` column would be converted to a `Map` in Arrow, but actually it's converted to a list column. This is not the correct behaviour in a lot of cases, because a named list in R is often treated more like a map/dictionary than a list:

```
> as_arrow_table(x)
Table
3 rows x 2 columns
$id
$metadata: list>>
```

Secondly, even if I manually request arrow to use a map column, it fails:

```r
> arrow_table(x, schema = schema(id=int32(), metadata = map_of(string(), int32())))
Error: Invalid: Can only convert data frames to Struct type
```

It would be wonderful if one or both of these pathways worked, to allow named lists to be serialized as arrow maps.

Session info below:

```R
> sessionInfo()
R version 4.2.1 (2022-06-23)
Platform: x86_64-pc-linux-gnu (64-bit)
Running under: CentOS Linux 7 (Core)

Matrix products: default
BLAS: /stornext/System/data/apps/R/R-4.2.1/lib64/R/lib/libRblas.so
LAPACK: /stornext/System/data/apps/R/R-4.2.1/lib64/R/lib/libRlapack.so

locale:
[1] LC_CTYPE=en_AU.UTF-8 LC_NUMERIC=C
[3] LC_TIME=en_AU.UTF-8 LC_COLLATE=en_AU.UTF-8
[5] LC_MONETARY=en_AU.UTF-8 LC_MESSAGES=en_AU.UTF-8
[7] LC_PAPER=en_AU.UTF-8 LC_NAME=C
[9] LC_ADDRESS=C LC_TELEPHONE=C
[11] LC_MEASUREMENT=en_AU.UTF-8 LC_IDENTIFICATION=C

attached base packages:
[1] stats graphics grDevices utils datasets methods base

other attached packages:
[1] tibble_3.1.8 arrow_11.0.0.2

loaded via a namespace (and not attached):
[1] fansi_1.0.3 utf8_1.2.2 assertthat_0.2.1 R6_2.5.1
[5] lifecycle_1.0.3 magrittr_2.0.3 pillar_1.8.1 rlang_1.0.6
[9] cli_3.6.0 vctrs_0.5.2 bit64_4.0.5 glue_1.6.2
[13] purrr_1.0.1 bit_4.0.5 compiler_4.2.1 pkgconfig_2.0.3
[17] tidyselect_1.2.0
```

### Component(s)

R

Contributor guide

Open the contributing guide

Research direction

Reproduce the named-list example with tibble::tibble, as_arrow_table(), and arrow_table() using map_of(string(), int32()). Start by tracing how these entry points infer or apply the requested map schema. Done means named lists serialize as Arrow maps, either by inference or when the schema is supplied, with the reported conversion error covered by tests.

Written by the indexing model from the issue text.

Assessment

Tech stack
r
Domain
data
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.