apache / apache/arrow

[R] existing_data_behavior = "overwrite" does not work as expected in arrow::write_dataset

Open
#37,760 1 comment 0 reactions 0 assignees View on GitHub
Component: R Priority: Critical Type: bug
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 13h
Merged PRs (30d)
88

Description

### Describe the bug, including details regarding any error messages, version, and platform.

It seems that the default for existing_data_behavior='overwrite' does not actually overwrite existing data. @jonkeane

``` r
library(dplyr)
#>
#> Attaching package: 'dplyr'
#> The following objects are masked from 'package:stats':
#>
#> filter, lag
#> The following objects are masked from 'package:base':
#>
#> intersect, setdiff, setequal, union

path <- file.path(tempdir(), "mtcars-data")

mtcars %>%
group_by(cyl) %>%
arrow::write_dataset(path)

list.dirs(path, full.names = F)
#> [1] "" "cyl=4" "cyl=6" "cyl=8"
fs::file_size(path)
#> 160

mtcars %>%
group_by(vs) %>%
arrow::write_dataset(path, existing_data_behavior = "overwrite")

list.dirs(path, full.names = F)
#> [1] "" "cyl=4" "cyl=6" "cyl=8" "vs=0" "vs=1"
fs::file_size(path)
#> 224

sessionInfo()
#> R version 4.3.1 (2023-06-16)
#> Platform: aarch64-apple-darwin20 (64-bit)
#> Running under: macOS Ventura 13.5.2
#>
#> Matrix products: default
#> BLAS: /Library/Frameworks/R.framework/Versions/4.3-arm64/Resources/lib/libRblas.0.dylib
#> LAPACK: /Library/Frameworks/R.framework/Versions/4.3-arm64/Resources/lib/libRlapack.dylib; LAPACK version 3.11.0
#>
#> locale:
#> [1] en_US.UTF-8/en_US.UTF-8/en_US.UTF-8/C/en_US.UTF-8/en_US.UTF-8
#>
#> time zone: America/Chicago
#> tzcode source: internal
#>
#> attached base packages:
#> [1] stats graphics grDevices utils datasets methods base
#>
#> other attached packages:
#> [1] dplyr_1.1.3
#>
#> loaded via a namespace (and not attached):
#> [1] vctrs_0.6.3 cli_3.6.1 knitr_1.44 rlang_1.1.1
#> [5] xfun_0.40 purrr_1.0.2 generics_0.1.3 assertthat_0.2.1
#> [9] bit_4.0.5 glue_1.6.2 htmltools_0.5.6 fansi_1.0.4
#> [13] rmarkdown_2.24 evaluate_0.21 tibble_3.2.1 fastmap_1.1.1
#> [17] yaml_2.3.7 lifecycle_1.0.3 compiler_4.3.1 fs_1.6.3
#> [21] pkgconfig_2.0.3 rstudioapi_0.15.0 digest_0.6.33 R6_2.5.1
#> [25] reprex_2.0.2 tidyselect_1.2.0 utf8_1.2.3 pillar_1.9.0
#> [29] magrittr_2.0.3 bit64_4.0.5 tools_4.3.1 withr_2.5.0
#> [33] arrow_13.0.0
```

Created on 2023-09-17 with [reprex v2.0.2](https://reprex.tidyverse.org)

### Component(s)

R

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.