apache / apache/arrow

[R] passing a schema calls open_dataset to fail on hive-partitioned csv files

Open
#31,312 5 comments 0 reactions 0 assignees View on GitHub
Component: R Type: task
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 18h
Merged PRs (30d)
91

Description

Consider this reprex:

 

Create a dataset with hive partitions in csv format with write_dataset() (so cool!):

 
```java

library(arrow)
library(dplyr)
path <- fs::dir_create("tmp")
mtcars %>% group_by(gear) %>% write_dataset(path, format="csv")## works fine, even with 'collect()'
ds <- open_dataset(path, format="csv")## but pass a schema, and things fail
df <- open_dataset(path, format="csv", schema = ds$schema, skip_rows=1)
df %>% collect()

```
In the first call to open_dataset, we don't pass a schema and things work as expected. 

However, csv files often need a schema to be read in correctly, particularly with partitioned data where it is easy to 'guess' the wrong type.  Passing the schema though confuses open_dataset, because the grouping column (partition column) isn't found on the individual files even though it is mentioned in the schema!

Nor can we just omit the grouping column from the schema, since then it is effectively lost from the data. 

**Reporter**: [Carl Boettiger](https://issues.apache.org/jira/browse/ARROW-15879) / @cboettig
#### PRs and other links:
- [GitHub Pull Request #12831](https://github.com/apache/arrow/pull/12831)

**Note**: *This issue was originally created as [ARROW-15879](https://issues.apache.org/jira/browse/ARROW-15879). Please see the [migration documentation](https://github.com/apache/arrow/issues/14542) for further details.*

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the R reprex with write_dataset(), open_dataset(), and collect() on hive-partitioned CSV files. Review the linked pull request for the existing work. Done means passing an explicit schema succeeds while retaining the partition column and allowing collect() to complete.

Written by the indexing model from the issue text.

Assessment

Tech stack
r
Domain
data-engineering
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Clearly specified
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.