[R][Python] Partitioned Datasets are not correctly written if partitions differ by case
- Dominant language
- C++
- Stars
- 17.1k
- Forks
- 4.3k
- Avg merge
- 3d 13h
- Merged PRs (30d)
- 88
Description
### Describe the bug, including details regarding any error messages, version, and platform.
With {arrow} 14.0.0, I was able to import a large number of CSVs, merge the schemas, establish a partitioning and write them with `arrow::write_dataset()`.
The dataset can be manipulated in-memory and queried without issues, but most queries fail once the dataset is read from disk. The number of rows is also different.
The issue arises when the partitioning variable differs just by capitalization.
On Windows the file system is case-insensitive but {arrow} treats partitions as different, so the Parquet files end up corrupted.
### R reprex
https://github.com/apache/arrow/issues/38804#issuecomment-1824165163
### Python reprex
```python
import pyarrow as pa
import pyarrow.dataset as ds
tbl = pa.table({
'var_1': ("arrow", "arrow", "arrow"),
'var_2': ("arrow", "arrow", "arroW")
})
part = ds.partitioning(
pa.schema([("var_2", pa.string())]),
flavor="hive"
)
ds.write_dataset(tbl, "dataset_output", format="parquet", partitioning=part)
ds_out = ds.dataset("dataset_output")
```
Only one partition is written:
```python
ds_out.files
# ['dataset_output/var_2=arroW/part-0.parquet']
```
```python
ds_out.to_table()
# pyarrow.Table
# var_1: string
# ----
# var_1: [["arrow"]]
```
This is `arrow::arrow_info()`:
``` r
arrow::arrow_info()
#> Arrow package version: 14.0.0
#>
#> Capabilities:
#>
#> acero TRUE
#> dataset TRUE
#> substrait FALSE
#> parquet TRUE
#> json TRUE
#> s3 TRUE
#> gcs TRUE
#> utf8proc TRUE
#> re2 TRUE
#> snappy TRUE
#> gzip TRUE
#> brotli TRUE
#> zstd TRUE
#> lz4 TRUE
#> lz4_frame TRUE
#> lzo FALSE
#> bz2 TRUE
#> jemalloc FALSE
#> mimalloc TRUE
#>
#> Arrow options():
#>
#> arrow.use_threads FALSE
#>
#> Memory:
#>
#> Allocator mimalloc
#> Current 0 bytes
#> Max 0 bytes
#>
#> Runtime:
#>
#> SIMD Level avx2
#> Detected SIMD Level avx2
#>
#> Build:
#>
#> C++ Library Version 14.0.0
#> C++ Compiler GNU
#> C++ Compiler Version 10.3.0
#> Git ID 2dcee3f82c6cf54b53a64729fd81840efa583244
```
### Component(s)
R
Contributor guide
Assessment
This issue has not been assessed yet.