[R] Named lists cannot be serialized to a map column
- Dominant language
- C++
- Stars
- 17.1k
- Forks
- 4.3k
- Avg merge
- 3d 18h
- Merged PRs (30d)
- 91
Description
### Describe the bug, including details regarding any error messages, version, and platform.
Let's start with a simple table that contains a named list in each row:
```R
> x = tibble::tibble(id=1:3, metadata=list(list(a=1), list(b=2, c=3), list(d=3, e=4)))
> x
# A tibble: 3 × 2
id metadata
1 1
2 2
3 3
```
From the outset, I would hope that the `metadata` column would be converted to a `Map` in Arrow, but actually it's converted to a list column. This is not the correct behaviour in a lot of cases, because a named list in R is often treated more like a map/dictionary than a list:
```
> as_arrow_table(x)
Table
3 rows x 2 columns
$id
$metadata: list>>
```
Secondly, even if I manually request arrow to use a map column, it fails:
```r
> arrow_table(x, schema = schema(id=int32(), metadata = map_of(string(), int32())))
Error: Invalid: Can only convert data frames to Struct type
```
It would be wonderful if one or both of these pathways worked, to allow named lists to be serialized as arrow maps.
Session info below:
```R
> sessionInfo()
R version 4.2.1 (2022-06-23)
Platform: x86_64-pc-linux-gnu (64-bit)
Running under: CentOS Linux 7 (Core)
Matrix products: default
BLAS: /stornext/System/data/apps/R/R-4.2.1/lib64/R/lib/libRblas.so
LAPACK: /stornext/System/data/apps/R/R-4.2.1/lib64/R/lib/libRlapack.so
locale:
[1] LC_CTYPE=en_AU.UTF-8 LC_NUMERIC=C
[3] LC_TIME=en_AU.UTF-8 LC_COLLATE=en_AU.UTF-8
[5] LC_MONETARY=en_AU.UTF-8 LC_MESSAGES=en_AU.UTF-8
[7] LC_PAPER=en_AU.UTF-8 LC_NAME=C
[9] LC_ADDRESS=C LC_TELEPHONE=C
[11] LC_MEASUREMENT=en_AU.UTF-8 LC_IDENTIFICATION=C
attached base packages:
[1] stats graphics grDevices utils datasets methods base
other attached packages:
[1] tibble_3.1.8 arrow_11.0.0.2
loaded via a namespace (and not attached):
[1] fansi_1.0.3 utf8_1.2.2 assertthat_0.2.1 R6_2.5.1
[5] lifecycle_1.0.3 magrittr_2.0.3 pillar_1.8.1 rlang_1.0.6
[9] cli_3.6.0 vctrs_0.5.2 bit64_4.0.5 glue_1.6.2
[13] purrr_1.0.1 bit_4.0.5 compiler_4.2.1 pkgconfig_2.0.3
[17] tidyselect_1.2.0
```
### Component(s)
R
Contributor guide
Research direction
Reproduce the named-list example with tibble::tibble, as_arrow_table(), and arrow_table() using map_of(string(), int32()). Start by tracing how these entry points infer or apply the requested map schema. Done means named lists serialize as Arrow maps, either by inference or when the schema is supplied, with the reported conversion error covered by tests.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- r
- Domain
- data
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100