[R] Add an argument to `open_csv_dataset()` to repair duplicated column names or ignore them?
- Dominant language
- C++
- Stars
- 17.1k
- Forks
- 4.3k
- Avg merge
- 3d 18h
- Merged PRs (30d)
- 91
Description
### Describe the enhancement requested
Hello, I'm just getting started with `arrow` so I hope this request makes sense.
I have a list of CSV files that have some duplicated column names. When I use `readr::read_csv()`, it automatically "repairs" those names to make them unique but `arrow::open_csv_dataset()` errors because of those duplicated column names:
``` r
packageVersion("arrow")
#> [1] '11.0.0.3'
file_location <- tempfile(fileext = ".csv")
test <- data.frame(x = 1, x = 2, check.names = FALSE)
write.csv(test, file_location)
readr::read_csv(file_location)
#> New names:
#> Rows: 1 Columns: 3
#> ── Column specification
#> ──────────────────────────────────────────────────────── Delimiter: "," dbl
#> (3): ...1, x...2, x...3
#> ℹ Use `spec()` to retrieve the full column specification for this data. ℹ
#> Specify the column types or set `show_col_types = FALSE` to quiet this message.
#> • `` -> `...1`
#> • `x` -> `x...2`
#> • `x` -> `x...3`
#> # A tibble: 1 × 3
#> ...1 x...2 x...3
#>
#> 1 1 1 2
arrow::open_csv_dataset(file_location)
#> Error in `open_dataset()`:
#> ! Invalid: Error creating dataset. Could not read schema from 'C:/Users/etienne/AppData/Local/Temp/RtmpYLHyDe/file3c7c3a407a21.csv': Could not open CSV input source 'C:/Users/etienne/AppData/Local/Temp/RtmpYLHyDe/file3c7c3a407a21.csv': Invalid: CSV file contained multiple columns named x. Is this a 'csv' file?
#> Backtrace:
#> ▆
#> 1. └─arrow (local) ``(sources = file_location, delim = ",")
#> 2. └─arrow::open_dataset(...)
#> 3. └─base::tryCatch(...)
#> 4. └─base (local) tryCatchList(expr, classes, parentenv, handlers)
#> 5. └─base (local) tryCatchOne(expr, names, parentenv, handlers[[1L]])
#> 6. └─value[[3L]](cond)
#> 7. └─arrow:::augment_io_error_msg(e, call, format = format)
#> 8. └─rlang::abort(msg, call = call)
```
I could ignore those names with `col_select = - c()`. However, those columns are not necessarily in all CSV files and if they're not then `open_csv_dataset()` also errors.
Would it be possible to add an argument to repair the column names or to ignore them (i.e not load them)? Or am I missing something obvious?
Thanks for your work!
### Component(s)
R
Contributor guide
Research direction
Start at the R open_csv_dataset() entry point and reproduce the example with duplicated CSV column names. Trace how col_select is handled across files, then define and verify whether the requested behavior repairs duplicate names, ignores them, or supports both options consistently.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- r
- Domain
- data
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100