apache / apache/arrow

[R] open_dataset() on long vec of URIs uses much more RAM & is much slower than on partition root.

Open
#35,715 3 comments 0 reactions 0 assignees View on GitHub
Component: R Priority: Critical Type: bug
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 18h
Merged PRs (30d)
91

Description

### Describe the bug, including details regarding any error messages, version, and platform.

Using `open_dataset()` on a remote S3 root with lots of partition files can be quite slow just because listing files on S3 is really slow (https://github.com/apache/arrow/issues/34145). Some of this might be improved by https://github.com/apache/arrow/issues/34213, but apparently this slow listing is a well-known limitation of the S3 API, and Amazon provides the [S3 Inventory](https://docs.aws.amazon.com/AmazonS3/latest/userguide/storage-inventory.html) system precisely because of this limitation. This allows us to determine the URIs to each partition ahead-of-time, which should technically be way faster.

However, passing a large vector of URIs turns out to be even slower and less RAM-efficient! I'm not sure what's going on, but it feels to me like maybe `unify_schemas = FALSE` is being ignored, despite the docs saying it is the default setting for a vector of URIs.

Here's what should be a reproducible illustration of the issue.

```r
library(arrow)

s3 <- s3_bucket("neon4cast-scores/parquet/aquatics", endpoint_override = "data.ecoforecast.org", anonymous=TRUE)

bench::bench_time( # very slow
ds <- open_dataset(s3)
)

# Can we work around this with pre-computed vector of URIs?

bench::bench_time( # very slow, but available via S3 Inventory
all_paths <- s3$ls(recursive=TRUE)
)

all_paths <- all_paths[grepl("[.]parquet", all_paths)]
uris <- paste0("s3://neon4cast-scores/parquet/aquatics/", all_paths, "?endpoint_override=data.ecoforecast.org")

# should be fast now that we know the URIs ahead of time and avoid the ls() overhead. but wow this is worse!
bench::bench_time( # incredibly slow & accumulates much higher RAM use
open_dataset(uris)
)
````

(Also, note that duckdb can open this vector of URIs considerably more quickly, if with rather more verbose code.)

Not sure what I am missing here. Am I wrong in thinking that things should be faster using the pre-computed vector of URIs rather than leaving arrow to effectively have to do the recursive `ls` itself? Any idea what makes opening the vector of URIs so slow here? Is there any better alternative strategy in this setting (other than 'use fewer partitions', I know that would help but can't do so here). (@westonpace or others probably have some good insight here!)

### Component(s)

R

Contributor guide

Open the contributing guide

Research direction

Start with the R entry points shown in the report: open_dataset(), s3_bucket(), and s3$ls(). Reproduce the benchmark with the partition root and the precomputed URI vector, measuring elapsed time and RAM. Done means identifying why URI-based opening is slower or less memory-efficient and documenting or correcting the behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, r
Domain
data, performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.