[R] unify_schemas=FALSE does not improve open_dataset() read times
- Dominant language
- C++
- Stars
- 17.1k
- Forks
- 4.3k
- Avg merge
- 3d 18h
- Merged PRs (30d)
- 91
Description
open_dataset() provides the very helpful optional argument to set unify_schemas=FALSE, which should allow arrow to inspect a single parquet file instead of touching potentially thousands or more parquet files to determine a consistent unified schema. This ought to provide a substantial performance increase in contexts where the schema is known in advance.
Unfortunately, in my tests it seems to have no impact on performance. Consider the following reprexes:
default, unify_schemas=TRUE
```java
library(arrow)
ex <- s3_bucket("neon4cast-scores/parquet/terrestrial_30min", endpoint_override = "data.ecoforecast.org", anonymous=TRUE)
bench::bench_time(
{ open_dataset(ex) }
)
```
about 32 seconds for me.
manual, unify_schemas=FALSE:
```java
bench::bench_time({
open_dataset(ex, unify_schemas = FALSE)
})
```
takes about 32 seconds as well.
**Reporter**: [Carl Boettiger](https://issues.apache.org/jira/browse/ARROW-18114) / @cboettig
**Note**: *This issue was originally created as [ARROW-18114](https://issues.apache.org/jira/browse/ARROW-18114). Please see the [migration documentation](https://github.com/apache/arrow/issues/14542) for further details.*
Contributor guide
Research direction
Start with the R open_dataset() entry point and reproduce the two bench::bench_time examples against the S3 parquet dataset. Trace how unify_schemas=FALSE selects schema discovery, then verify that it avoids touching the full dataset and improves the reported read time without changing results.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- r
- Domain
- data-engineering, performance
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100