[R] read_parquet from s3 is slow and often flakey
- Dominant language
- C++
- Stars
- 17.1k
- Forks
- 4.3k
- Avg merge
- 3d 18h
- Merged PRs (30d)
- 91
Description
### Describe the bug, including details regarding any error messages, version, and platform.
R version 4.3.0
Ubuntu 20.04.6
Sometimes it fails, other times it hangs, but it always takes longer than similar methods using aws.s3 and it's unclear why. Here are some examples of the errors I see. Otherwise it's just a significant delay like it's running into a time out somewhere and retrying:
```
Error: IOError: AWS Error NETWORK_CONNECTION during GetObject operation: curlCode: 28, Timeout was reached
```
```
Error in url(file, open = "rb") : URL scheme unsupported by this method
```
## Reprex:
The two scripts below, in my example point to a small 5mb file. I can run this on a machine with arrow 8.0.0 or a machine with arrow 12.0.0 and the result is the same (significant delay with s3:// addresses)
```
library(dotenv)
library(tictoc)
load_dot_env()
tic(); df <- read_parquet("s3://somefile.parquet"); toc()
# 70.211 sec elapsed
```
vs.
```
library(aws.s3)
library(dotenv)
load_dot_env()
load_aws <- function(f) {
read_parquet(
get_object(
object = f,
bucket = "s3://bucket-name",
region = "eu-west-1"
)
)
}
tic(); df <- load_aws("s3://somefile.parquet"); toc()
# 1.394 sec elapsed
```
On the same machines I can run a similar test in Python and the delay is not there leading me to believe it is something explicitly with the implementation in R. Happy to help debug! Tell me what you need.
### Component(s)
R
Contributor guide
Research direction
Start by reproducing the delay and failures at the R read_parquet entry point with the supplied s3:// example, comparing it with the aws.s3 get_object path. Trace the S3 request and timeout behavior, then verify that direct reads are reliable and approach the reported timing without regressions across the stated Arrow versions and Ubuntu environment.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, r
- Domain
- cloud, data
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 32/100