[R] existing_data_behavior = "overwrite" does not work as expected in arrow::write_dataset
- Dominant language
- C++
- Stars
- 17.1k
- Forks
- 4.3k
- Avg merge
- 3d 13h
- Merged PRs (30d)
- 88
Description
### Describe the bug, including details regarding any error messages, version, and platform.
It seems that the default for existing_data_behavior='overwrite' does not actually overwrite existing data. @jonkeane
``` r
library(dplyr)
#>
#> Attaching package: 'dplyr'
#> The following objects are masked from 'package:stats':
#>
#> filter, lag
#> The following objects are masked from 'package:base':
#>
#> intersect, setdiff, setequal, union
path <- file.path(tempdir(), "mtcars-data")
mtcars %>%
group_by(cyl) %>%
arrow::write_dataset(path)
list.dirs(path, full.names = F)
#> [1] "" "cyl=4" "cyl=6" "cyl=8"
fs::file_size(path)
#> 160
mtcars %>%
group_by(vs) %>%
arrow::write_dataset(path, existing_data_behavior = "overwrite")
list.dirs(path, full.names = F)
#> [1] "" "cyl=4" "cyl=6" "cyl=8" "vs=0" "vs=1"
fs::file_size(path)
#> 224
sessionInfo()
#> R version 4.3.1 (2023-06-16)
#> Platform: aarch64-apple-darwin20 (64-bit)
#> Running under: macOS Ventura 13.5.2
#>
#> Matrix products: default
#> BLAS: /Library/Frameworks/R.framework/Versions/4.3-arm64/Resources/lib/libRblas.0.dylib
#> LAPACK: /Library/Frameworks/R.framework/Versions/4.3-arm64/Resources/lib/libRlapack.dylib; LAPACK version 3.11.0
#>
#> locale:
#> [1] en_US.UTF-8/en_US.UTF-8/en_US.UTF-8/C/en_US.UTF-8/en_US.UTF-8
#>
#> time zone: America/Chicago
#> tzcode source: internal
#>
#> attached base packages:
#> [1] stats graphics grDevices utils datasets methods base
#>
#> other attached packages:
#> [1] dplyr_1.1.3
#>
#> loaded via a namespace (and not attached):
#> [1] vctrs_0.6.3 cli_3.6.1 knitr_1.44 rlang_1.1.1
#> [5] xfun_0.40 purrr_1.0.2 generics_0.1.3 assertthat_0.2.1
#> [9] bit_4.0.5 glue_1.6.2 htmltools_0.5.6 fansi_1.0.4
#> [13] rmarkdown_2.24 evaluate_0.21 tibble_3.2.1 fastmap_1.1.1
#> [17] yaml_2.3.7 lifecycle_1.0.3 compiler_4.3.1 fs_1.6.3
#> [21] pkgconfig_2.0.3 rstudioapi_0.15.0 digest_0.6.33 R6_2.5.1
#> [25] reprex_2.0.2 tidyselect_1.2.0 utf8_1.2.3 pillar_1.9.0
#> [29] magrittr_2.0.3 bit64_4.0.5 tools_4.3.1 withr_2.5.0
#> [33] arrow_13.0.0
```
Created on 2023-09-17 with [reprex v2.0.2](https://reprex.tidyverse.org)
### Component(s)
R
Contributor guide
Assessment
This issue has not been assessed yet.