apache / apache/arrow-nanoarrow

[R] Convert arrow dictionary to R factor via `as.data.frame.nanoarrow_array_stream()`?

Open
#513 3 comments 0 reactions 0 assignees View on GitHub
Dominant language
C
Stars
249
Forks
68
Avg merge
2d 21h
Merged PRs (30d)
11

Description

Maybe related to #220

I noticed that if we convert nanoarrow_array_stream to data.frame, the dictionary becomes a character.

``` r
stream <- data.frame(
x = as.factor(letters[1:5]),
y = as.factor(1:5)
) |>
nanoarrow::as_nanoarrow_array_stream()

stream
#> , y: dictionary(int32)>>
#> $ get_schema:function ()
#> $ get_next :function (schema = x$get_schema(), validate = TRUE)
#> $ release :function ()
```

``` r

stream |>
tibble::as_tibble()
#> # A tibble: 5 × 2
#> x y
#>
#> 1 a 1
#> 2 b 2
#> 3 c 3
#> 4 d 4
#> 5 e 5
```

Created on 2024-06-09 with [reprex v2.1.0](https://reprex.tidyverse.org)

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reproducing the conversion shown in the issue through as.data.frame.nanoarrow_array_stream() and compare the resulting columns with the original factors. Trace the dictionary handling in that entry point; done means dictionary-encoded columns are returned as R factors rather than characters, with coverage for both example columns.

Written by the indexing model from the issue text.

Assessment

Tech stack
r
Domain
data
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.