apache / apache/arrow

[R] open_dataset partitioning fails when source is a character vector of paths

Open
#41,146 2 comments 0 reactions 0 assignees View on GitHub
Component: R Type: bug
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 13h
Merged PRs (30d)
88

Description

### Describe the bug, including details regarding any error messages, version, and platform.

When using R, `open_dataset()` does not correctly parse partitions if the `sources` argument is a character vector of paths to individual files. However, it does work fine in pyarrow.

My particular use case is I have a folder structure like `x/y/{a,b,c}.parquet`, where the a, b, c files are all different types of files. That is, I only want to load a.parquet files together, b.parquet files together etc. (Arguably, the better structure for arrow here would be to create a, b, c folders before the partitions...)

If I produce the list of files, I can load them in fine with `open_dataset`, but the partitions are not loaded.

As a minimal example:

``` r
library(palmerpenguins)
library(dplyr)
library(arrow)

data(penguins)

td <- withr::local_tempdir("penguins")

penguins |>
group_by(species) |>
arrow::write_dataset(
format = "parquet",
td,
hive_style = TRUE
)

# this works fine, by passing in the folder
open_dataset(td, partitioning = c("species")) |>
count(species) |>
collect()

# does not work
td |>
fs::dir_ls(recurse = TRUE, glob = "*.parquet") |>
open_dataset(
partitioning = "species"
) |>
count(species) |>
collect()

# as per docs, fails
td |>
fs::dir_ls(recurse = TRUE, glob = "*.parquet") |>
open_dataset(
partitioning = "species",
factory_options = list(
partition_base_dir = td
)
) |>
count(species) |>
collect()

# does not work
with_dir(
td,
{
fs::dir_ls(".", recurse = TRUE, glob = "*.parquet") |>
open_dataset(
partitioning = "species"
) |>
count(species) |>
collect()
}
)
```

This works fine in python though

``` python
import pyarrow as pa
import pyarrow.dataset as ds
import os

path = "penguins"
files = [
f"{path}/{i}/{j}"
for i in os.listdir(path)
for j in os.listdir(f"{path}/{i}")
]

(
ds.dataset(
files,
partition_base_dir = path,
partitioning = ds.partitioning(
pa.schema([
("species", pa.string())
])
)
)
.to_table()
.to_pandas()
.value_counts("species")
)
```

### Component(s)

R

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.