[R] Error when reading parquet file using FileSystem object
- Dominant language
- C++
- Stars
- 17.1k
- Forks
- 4.3k
- Avg merge
- 3d 13h
- Merged PRs (30d)
- 88
Description
Hello,
I am encountering an issue when trying to read a parquet file using `read_parquet` with an `S3FileSystem` created with `s3_bucket()`.
I created a worker that get the last parquet file id uploaded to an S3 bucket (using S3 api) and then trying to read the file by calling the `$path()` method with my filename as the arg. I created a custom function `read_table_fromS3` to do this.
This errors occurs after a second call to `read_parquet()` when running my script with `Rscript` within a docker container built with all dependencies needed and access to AWS credentials:
The error I catch :
```
Error in fs___FileSystem__OpenInputFile(self, clean_path_rel(path)) :
ignoring SIGPIPE signal
Calls: read_table_fromS3 ... make_readable_file -> -> fs___FileSystem__OpenInputFile
```
Here is a (non reproducible) sample of what my script is doing :
```
library(arrow)
s3target <- s3_bucket(my_aws_s3_bucket,
access_key = my_aws_access_key,
secret_key = my_aws_secret_key
)
# oversimplified version of my actual function
read_table_fromS3 <- function(s3bucket, id) {
file_path <- paste0(id, ".parquet")
path <- s3bucket$path(file_path)
read_parquet(path)
}
# first read_parquet() call : OK
read_table_fromS3(s3target, )
# get last file identifier
last_file_id <- get_last_file_id(...)
#
# instructions to test if last_file_id is the latest available (can take several minutes/hours)
#
# second read_parquet() call : KO
read_table_fromS3(s3target, last_file_id)
> Error in fs___FileSystem__OpenInputFile(self, clean_path_rel(path)) :
ignoring SIGPIPE signal
```
Here is the `sessionInfo()` from my running container :
```
> sessionInfo()
R version 4.1.1 (2021-08-10)
Platform: x86_64-pc-linux-gnu (64-bit)
Running under: Ubuntu 20.04.3 LTS
Matrix products: default
BLAS/LAPACK: /usr/lib/x86_64-linux-gnu/openblas-pthread/libopenblasp-r0.3.8.so
locale:
[1] LC_CTYPE=en_US.UTF-8 LC_NUMERIC=C
[3] LC_TIME=en_US.UTF-8 LC_COLLATE=en_US.UTF-8
[5] LC_MONETARY=en_US.UTF-8 LC_MESSAGES=C
[7] LC_PAPER=en_US.UTF-8 LC_NAME=C
[9] LC_ADDRESS=C LC_TELEPHONE=C
[11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C
attached base packages:
[1] stats graphics grDevices utils datasets methods base
other attached packages:
[1] optparse_1.7.1 googlesheets4_1.0.0 lubridate_1.8.0
[4] futile.logger_1.4.3 stringr_1.4.0 arrow_6.0.1
[7] paws.storage_0.1.12 data.table_1.14.2 here_1.0.1
[10] glue_1.6.0
loaded via a namespace (and not attached):
[1] cellranger_1.1.0 compiler_4.1.1 pillar_1.6.4
[4] formatR_1.11 futile.options_1.0.1 tools_4.1.1
[7] digest_0.6.29 bit_4.0.4 googledrive_2.0.0
[10] jsonlite_1.7.2 gargle_1.2.0 lifecycle_1.0.1
[13] tibble_3.1.6 pkgconfig_2.0.3 rlang_0.4.12
[16] yaml_2.2.1 curl_4.3.2 xml2_1.3.3
[19] httr_1.4.2 askpass_1.1 generics_0.1.1
[22] fs_1.5.2 vctrs_0.3.8 rprojroot_2.0.2
[25] bit64_4.0.5 tidyselect_1.1.1 getopt_1.20.3
[28] R6_2.5.1 fansi_0.5.0 paws.common_0.3.15
[31] purrr_0.3.4 lambda.r_1.2.4 magrittr_2.0.1
[34] ellipsis_0.3.2 assertthat_0.2.1 config_0.3.1
[37] utf8_1.2.2 stringi_1.7.6 openssl_1.4.6
[40] crayon_1.4.2
```
Contributor guide
Research direction
Start with the R entry points `s3_bucket()`, `$path()`, and `read_parquet()`, then trace the second call to `FileSystem__OpenInputFile` in the reported Docker environment. Reproduce the two-read sequence if possible; done when the repeated read succeeds or the SIGPIPE failure is isolated to a specific component.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, r
- Domain
- cloud, data
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100