[R] open_dataset partitioning fails when source is a character vector of paths
- Dominant language
- C++
- Stars
- 17.1k
- Forks
- 4.3k
- Avg merge
- 3d 13h
- Merged PRs (30d)
- 88
Description
### Describe the bug, including details regarding any error messages, version, and platform.
When using R, `open_dataset()` does not correctly parse partitions if the `sources` argument is a character vector of paths to individual files. However, it does work fine in pyarrow.
My particular use case is I have a folder structure like `x/y/{a,b,c}.parquet`, where the a, b, c files are all different types of files. That is, I only want to load a.parquet files together, b.parquet files together etc. (Arguably, the better structure for arrow here would be to create a, b, c folders before the partitions...)
If I produce the list of files, I can load them in fine with `open_dataset`, but the partitions are not loaded.
As a minimal example:
``` r
library(palmerpenguins)
library(dplyr)
library(arrow)
data(penguins)
td <- withr::local_tempdir("penguins")
penguins |>
group_by(species) |>
arrow::write_dataset(
format = "parquet",
td,
hive_style = TRUE
)
# this works fine, by passing in the folder
open_dataset(td, partitioning = c("species")) |>
count(species) |>
collect()
# does not work
td |>
fs::dir_ls(recurse = TRUE, glob = "*.parquet") |>
open_dataset(
partitioning = "species"
) |>
count(species) |>
collect()
# as per docs, fails
td |>
fs::dir_ls(recurse = TRUE, glob = "*.parquet") |>
open_dataset(
partitioning = "species",
factory_options = list(
partition_base_dir = td
)
) |>
count(species) |>
collect()
# does not work
with_dir(
td,
{
fs::dir_ls(".", recurse = TRUE, glob = "*.parquet") |>
open_dataset(
partitioning = "species"
) |>
count(species) |>
collect()
}
)
```
This works fine in python though
``` python
import pyarrow as pa
import pyarrow.dataset as ds
import os
path = "penguins"
files = [
f"{path}/{i}/{j}"
for i in os.listdir(path)
for j in os.listdir(f"{path}/{i}")
]
(
ds.dataset(
files,
partition_base_dir = path,
partitioning = ds.partitioning(
pa.schema([
("species", pa.string())
])
)
)
.to_table()
.to_pandas()
.value_counts("species")
)
```
### Component(s)
R
Contributor guide
Assessment
This issue has not been assessed yet.