[R] cannot mutate into a hive-partition key
- Dominant language
- C++
- Stars
- 17.1k
- Forks
- 4.3k
- Avg merge
- 3d 13h
- Merged PRs (30d)
- 88
Description
### Describe the bug, including details regarding any error messages, version, and platform.
There are use-cases where I cannot use the "normal" filtering with hive-partitioning and must `open_dataset` directly on a subdir. Using `add_filename()`, it is in general not difficult to infer the string value of the key variable, but ... I cannot assign it directly in a lazy `mutate`.
```r
td <- tempfile(fileext=".d")
dir.create(td)
arrow::write_dataset(mtcars, td, partitioning="cyl")
list.files(td, recursive = TRUE, full.names = TRUE)
# [1] "/home/r2/tmp/RtmpsoiQjF/file640d75e7cb0d4.d/cyl=4/part-0.parquet" "/home/r2/tmp/RtmpsoiQjF/file640d75e7cb0d4.d/cyl=6/part-0.parquet"
# [3] "/home/r2/tmp/RtmpsoiQjF/file640d75e7cb0d4.d/cyl=8/part-0.parquet"
arrow::open_dataset(file.path(td, "cyl=6")) |>
mutate(
.fn = add_filename(),
cyl = sub(".*/cyl=([^/]*)/.*", "\\1", .fn),
cyl = if_else(cyl == .fn, NA_character_, cyl)
) |>
collect()
# Error in compute.arrow_dplyr_query(x) : ℹ In index: 2.
# Caused by error in `x$Equals()`:
# ! attempt to apply non-function
arrow::open_dataset(file.path(td, "cyl=6")) |>
mutate(
.fn = add_filename(),
.cyl = sub(".*/cyl=([^/]*)/.*", "\\1", .fn),
.cyl = if_else(.cyl == .fn, NA_character_, .cyl)
) |>
collect()
# # A tibble: 7 × 12
# mpg disp hp drat wt qsec vs am gear carb .fn .cyl
#
# 1 21 160 110 3.9 2.62 16.5 0 1 4 4 /home/r2/tmp/RtmpsoiQjF/file640d75e7cb0d4.d/cyl=6/part-0.parquet 6
# 2 21 160 110 3.9 2.88 17.0 0 1 4 4 /home/r2/tmp/RtmpsoiQjF/file640d75e7cb0d4.d/cyl=6/part-0.parquet 6
# 3 21.4 258 110 3.08 3.22 19.4 1 0 3 1 /home/r2/tmp/RtmpsoiQjF/file640d75e7cb0d4.d/cyl=6/part-0.parquet 6
# 4 18.1 225 105 2.76 3.46 20.2 1 0 3 1 /home/r2/tmp/RtmpsoiQjF/file640d75e7cb0d4.d/cyl=6/part-0.parquet 6
# 5 19.2 168. 123 3.92 3.44 18.3 1 0 4 4 /home/r2/tmp/RtmpsoiQjF/file640d75e7cb0d4.d/cyl=6/part-0.parquet 6
# 6 17.8 168. 123 3.92 3.44 18.9 1 0 4 4 /home/r2/tmp/RtmpsoiQjF/file640d75e7cb0d4.d/cyl=6/part-0.parquet 6
# 7 19.7 145 175 3.62 2.77 15.5 0 1 5 6 /home/r2/tmp/RtmpsoiQjF/file640d75e7cb0d4.d/cyl=6/part-0.parquet 6
```
The workaround is to rename it, but ... if I need to do this renaming and still return a lazy table, then this is a hard-fail.
The question then could be an either/or question. Is it possible to:
1. Directly return the key value in the data when opening a directory that is using hive partitioning? Or ...
2. Assign to a variable that is never seen or available in the object or data anyway.
The fact that in this case `cyl` can be found nowhere in the attributes confounded me for a long time until I realized it was the key itself (and not the literal `cyl`).
(For the record, it is not an option to return the filename and expect the caller to do the parsing, nor is it good to have a renamed keyfield in the data.)
Thanks!
### Component(s)
R
Contributor guide
Assessment
This issue has not been assessed yet.