[C++] Can't open partitioned dataset if the root directory has "=" in its name
- Dominant language
- C++
- Stars
- 17.1k
- Forks
- 4.3k
- Avg merge
- 3d 13h
- Merged PRs (30d)
- 88
Description
Not sure if this is a bug or "just how Hive style partitioning works" but if I try to open a dataset where the root directory has an "=" in it, I have to specify that directory in my partitioning to be able to successfully open it.
This has caused users to trip up when they've saved one directory from a partitioned dataset somewhere and tried to then open this directory as a dataset.
```r
library(arrow)
td <- tempfile()
dir.create(td)
# directory with equals sign in name
subdir <- file.path(td, "foo=bar")
dir.create(subdir)
write_dataset(mtcars, subdir, partitioning = "am")
list.files(td, recursive = TRUE)
#> [1] "foo=bar/am=0/part-0.parquet" "foo=bar/am=1/part-0.parquet"
# doesn't work
open_dataset(subdir, partitioning = "am")
#> Error:
#> ! "partitioning" does not match the detected Hive-style partitions: c("foo", "am")
#> ℹ Omit "partitioning" to use the Hive partitions
#> ℹ Set `hive_style = FALSE` to override what was detected
#> ℹ Or, to rename partition columns, call `select()` or `rename()` after opening the dataset
# works
open_dataset(subdir, partitioning = c("foo", "am"))
#> FileSystemDataset with 2 Parquet files
#> mpg: double
#> cyl: double
#> disp: double
#> hp: double
#> drat: double
#> wt: double
#> qsec: double
#> vs: double
#> gear: double
#> carb: double
#> foo: string
#> am: int32
#>
#> See $metadata for additional Schema metadata
```
Compare this with the same example but the folder is just called "foobar" instead of "foo=bar".
```r
td <- tempfile()
dir.create(td)
subdir <- file.path(td, "foobar")
dir.create(subdir)
write_dataset(mtcars, subdir, partitioning = "am")
list.files(td, recursive = TRUE)
#> [1] "foobar/am=0/part-0.parquet" "foobar/am=1/part-0.parquet"
# works
open_dataset(subdir, partitioning = "am")
#> FileSystemDataset with 2 Parquet files
#> mpg: double
#> cyl: double
#> disp: double
#> hp: double
#> drat: double
#> wt: double
#> qsec: double
#> vs: double
#> gear: double
#> carb: double
#> am: int32
#>
#> See $metadata for additional Schema metadata
```
**Reporter**: [Nicola Crane](https://issues.apache.org/jira/browse/ARROW-15880) / @thisisnic
**Note**: *This issue was originally created as [ARROW-15880](https://issues.apache.org/jira/browse/ARROW-15880). Please see the [migration documentation](https://github.com/apache/arrow/issues/14542) for further details.*
Contributor guide
Research direction
Start by reproducing the R example with open_dataset(), write_dataset(), and a root directory named "foo=bar"; compare it with the "foobar" case. Trace how open_dataset() detects Hive-style partitions and interprets the supplied partitioning names. Done means the partitioned dataset opens with partitioning = "am" without treating the root directory name as a partition column.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp, r
- Domain
- data-engineering, databases
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100