[R] passing a schema calls open_dataset to fail on hive-partitioned csv files
- Dominant language
- C++
- Stars
- 17.1k
- Forks
- 4.3k
- Avg merge
- 3d 18h
- Merged PRs (30d)
- 91
Description
Consider this reprex:
Create a dataset with hive partitions in csv format with write_dataset() (so cool!):
```java
library(arrow)
library(dplyr)
path <- fs::dir_create("tmp")
mtcars %>% group_by(gear) %>% write_dataset(path, format="csv")## works fine, even with 'collect()'
ds <- open_dataset(path, format="csv")## but pass a schema, and things fail
df <- open_dataset(path, format="csv", schema = ds$schema, skip_rows=1)
df %>% collect()
```
In the first call to open_dataset, we don't pass a schema and things work as expected.
However, csv files often need a schema to be read in correctly, particularly with partitioned data where it is easy to 'guess' the wrong type. Passing the schema though confuses open_dataset, because the grouping column (partition column) isn't found on the individual files even though it is mentioned in the schema!
Nor can we just omit the grouping column from the schema, since then it is effectively lost from the data.
**Reporter**: [Carl Boettiger](https://issues.apache.org/jira/browse/ARROW-15879) / @cboettig
#### PRs and other links:
- [GitHub Pull Request #12831](https://github.com/apache/arrow/pull/12831)
**Note**: *This issue was originally created as [ARROW-15879](https://issues.apache.org/jira/browse/ARROW-15879). Please see the [migration documentation](https://github.com/apache/arrow/issues/14542) for further details.*
Contributor guide
Research direction
Start by reproducing the R reprex with write_dataset(), open_dataset(), and collect() on hive-partitioned CSV files. Review the linked pull request for the existing work. Done means passing an explicit schema succeeds while retaining the partition column and allowing collect() to complete.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- r
- Domain
- data-engineering
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Clearly specified
- Newbie friendliness
- 25/100