apache / apache/arrow

[R] Error in rawToChar(out) : long vectors not supported yet

Open
#35,382 9 comments 0 reactions 0 assignees View on GitHub
Component: R Status: needs champion Type: bug
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 18h
Merged PRs (30d)
91

Description

### Describe the bug, including details regarding any error messages, version, and platform.

I was trying to write long data with 1.2B rows and five variables from R.
It takes long time to raise an error:
```{r}
> write_parquet(taq_files, 'data/taq_files.parquet')
Error in rawToChar(out) : long vectors not supported yet: raw.c:68
```

SessionInfo:
```{r}
> sessionInfo()
R version 4.2.3 (2023-03-15)
Platform: x86_64-pc-linux-gnu (64-bit)
Running under: Ubuntu 22.04.2 LTS

Matrix products: default
BLAS: /usr/lib/x86_64-linux-gnu/openblas-pthread/libblas.so.3
LAPACK: /usr/lib/x86_64-linux-gnu/openblas-pthread/libopenblasp-r0.3.20.so

locale:
[1] LC_CTYPE=en_US.UTF-8 LC_NUMERIC=C LC_TIME=en_US.UTF-8 LC_COLLATE=en_US.UTF-8 LC_MONETARY=en_US.UTF-8
[6] LC_MESSAGES=en_US.UTF-8 LC_PAPER=en_US.UTF-8 LC_NAME=C LC_ADDRESS=C LC_TELEPHONE=C
[11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C

attached base packages:
[1] stats graphics grDevices utils datasets methods base

other attached packages:
[1] fstcore_0.9.14 fst_0.9.8 gmailr_1.0.1 qs_0.25.4 fasttime_1.1-0 arrow_11.0.0.3 forcats_1.0.0 stringr_1.5.0
[9] dplyr_1.1.0 purrr_1.0.1 readr_2.1.3 tidyr_1.3.0 tibble_3.1.8 ggplot2_3.4.0 tidyverse_1.3.2 data.table_1.14.8

loaded via a namespace (and not attached):
[1] Rcpp_1.0.10 lubridate_1.9.0 assertthat_0.2.1 utf8_1.2.3 R6_2.5.1 cellranger_1.1.0 backports_1.4.1
[8] reprex_2.0.2 httr_1.4.4 pillar_1.8.1 rlang_1.0.6 googlesheets4_1.0.1 curl_5.0.0 readxl_1.4.1
[15] rstudioapi_0.14 googledrive_2.0.0 bit_4.0.5 munsell_0.5.0 broom_1.0.3 compiler_4.2.3 modelr_0.1.10
[22] pkgconfig_2.0.3 base64enc_0.1-3 tidyselect_1.2.0 fansi_1.0.4 crayon_1.5.2 tzdb_0.3.0 dbplyr_2.3.2
[29] withr_2.5.0 rappdirs_0.3.3 grid_4.2.3 jsonlite_1.8.4 gtable_0.3.1 lifecycle_1.0.3 DBI_1.1.3
[36] pacman_0.5.1 magrittr_2.0.3 scales_1.2.1 RcppParallel_5.1.6 cli_3.6.0 stringi_1.7.12 fs_1.6.0
[43] xml2_1.3.3 ellipsis_0.3.2 generics_0.1.3 vctrs_0.5.2 stringfish_0.15.7 RApiSerialize_0.1.2 rematch2_2.1.2
[50] tools_4.2.3 bit64_4.0.5 glue_1.6.2 hms_1.1.2 parallel_4.2.3 timechange_0.2.0 colorspace_2.1-0
[57] gargle_1.2.1 rvest_1.0.3 haven_2.5.1
```

### Component(s)

R

Contributor guide

Open the contributing guide

Research direction

Start with the R entry point write_parquet and reproduce the reported rawToChar(out) failure using the 1.2B-row, five-variable input. Trace where the error originates; done means the same write completes without the reported long-vector error.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp, r
Domain
data-engineering
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.