apache / apache/arrow

[R] read_parquet from s3 is slow and often flakey

Open
#36,007 14 comments 0 reactions 0 assignees View on GitHub
Component: R Priority: Critical Type: bug
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 18h
Merged PRs (30d)
91

Description

### Describe the bug, including details regarding any error messages, version, and platform.

R version 4.3.0
Ubuntu 20.04.6

Sometimes it fails, other times it hangs, but it always takes longer than similar methods using aws.s3 and it's unclear why. Here are some examples of the errors I see. Otherwise it's just a significant delay like it's running into a time out somewhere and retrying:

```
Error: IOError: AWS Error NETWORK_CONNECTION during GetObject operation: curlCode: 28, Timeout was reached
```

```
Error in url(file, open = "rb") : URL scheme unsupported by this method
```

## Reprex:

The two scripts below, in my example point to a small 5mb file. I can run this on a machine with arrow 8.0.0 or a machine with arrow 12.0.0 and the result is the same (significant delay with s3:// addresses)

```
library(dotenv)
library(tictoc)

load_dot_env()

tic(); df <- read_parquet("s3://somefile.parquet"); toc()
# 70.211 sec elapsed
```

vs.

```
library(aws.s3)
library(dotenv)

load_dot_env()

load_aws <- function(f) {
read_parquet(
get_object(
object = f,
bucket = "s3://bucket-name",
region = "eu-west-1"
)
)
}

tic(); df <- load_aws("s3://somefile.parquet"); toc()
# 1.394 sec elapsed
```

On the same machines I can run a similar test in Python and the delay is not there leading me to believe it is something explicitly with the implementation in R. Happy to help debug! Tell me what you need.

### Component(s)

R

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the delay and failures at the R read_parquet entry point with the supplied s3:// example, comparing it with the aws.s3 get_object path. Trace the S3 request and timeout behavior, then verify that direct reads are reliable and approach the reported timing without regressions across the stated Arrow versions and Ubuntu environment.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, r
Domain
cloud, data
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
32/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.