apache / apache/arrow

[R] Add an argument to `open_csv_dataset()` to repair duplicated column names or ignore them?

Open
#34,965 10 comments 0 reactions 0 assignees View on GitHub
Component: R Status: needs champion Type: enhancement
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 18h
Merged PRs (30d)
91

Description

### Describe the enhancement requested

Hello, I'm just getting started with `arrow` so I hope this request makes sense.

I have a list of CSV files that have some duplicated column names. When I use `readr::read_csv()`, it automatically "repairs" those names to make them unique but `arrow::open_csv_dataset()` errors because of those duplicated column names:

``` r
packageVersion("arrow")
#> [1] '11.0.0.3'

file_location <- tempfile(fileext = ".csv")

test <- data.frame(x = 1, x = 2, check.names = FALSE)
write.csv(test, file_location)

readr::read_csv(file_location)
#> New names:
#> Rows: 1 Columns: 3
#> ── Column specification
#> ──────────────────────────────────────────────────────── Delimiter: "," dbl
#> (3): ...1, x...2, x...3
#> ℹ Use `spec()` to retrieve the full column specification for this data. ℹ
#> Specify the column types or set `show_col_types = FALSE` to quiet this message.
#> • `` -> `...1`
#> • `x` -> `x...2`
#> • `x` -> `x...3`
#> # A tibble: 1 × 3
#> ...1 x...2 x...3
#>
#> 1 1 1 2

arrow::open_csv_dataset(file_location)
#> Error in `open_dataset()`:
#> ! Invalid: Error creating dataset. Could not read schema from 'C:/Users/etienne/AppData/Local/Temp/RtmpYLHyDe/file3c7c3a407a21.csv': Could not open CSV input source 'C:/Users/etienne/AppData/Local/Temp/RtmpYLHyDe/file3c7c3a407a21.csv': Invalid: CSV file contained multiple columns named x. Is this a 'csv' file?
#> Backtrace:
#> ▆
#> 1. └─arrow (local) ``(sources = file_location, delim = ",")
#> 2. └─arrow::open_dataset(...)
#> 3. └─base::tryCatch(...)
#> 4. └─base (local) tryCatchList(expr, classes, parentenv, handlers)
#> 5. └─base (local) tryCatchOne(expr, names, parentenv, handlers[[1L]])
#> 6. └─value[[3L]](cond)
#> 7. └─arrow:::augment_io_error_msg(e, call, format = format)
#> 8. └─rlang::abort(msg, call = call)
```

I could ignore those names with `col_select = - c()`. However, those columns are not necessarily in all CSV files and if they're not then `open_csv_dataset()` also errors.

Would it be possible to add an argument to repair the column names or to ignore them (i.e not load them)? Or am I missing something obvious?

Thanks for your work!

### Component(s)

R

Contributor guide

Open the contributing guide

Research direction

Start at the R open_csv_dataset() entry point and reproduce the example with duplicated CSV column names. Trace how col_select is handled across files, then define and verify whether the requested behavior repairs duplicate names, ignores them, or supports both options consistently.

Written by the indexing model from the issue text.

Assessment

Tech stack
r
Domain
data
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.