[R][C++][Dataset] open_dataset and open_csv_dataset do not use skip argument
- Dominant language
- C++
- Stars
- 17.1k
- Forks
- 4.3k
- Avg merge
- 3d 13h
- Merged PRs (30d)
- 88
Description
### Describe the bug, including details regarding any error messages, version, and platform.
I noticed that a CSV with header rows can be successfully read with `read_csv_arrow(..., skip=N)`, but not with `open_csv_dataset` unless a schema is provided. An example is below.
I expected `open_csv_dataset` to use the skip argument the same way `read_csv_arrow` does, skipping a fixed number of rows from every CSV, but it seems to not be skipping any. I think this is likely a bug -- maybe in the schema parsing?
```R
library(arrow)
lines <- c(
"This line should be skipped",
"This one too, even though it has a comma",
"X,Y,Z",
"1,2,3",
"4,5,6"
)
tmp_dir <- file.path(tempdir(), "arrow_test")
dir.create(tmp_dir, recursive=TRUE)
tmp_csv <- file.path(tmp_dir, "test.csv")
writeLines(lines, tmp_csv)
# This works as expected, skipping the first two lines and reading headers from the third
read_csv_arrow(tmp_csv, skip=2L)
open_csv_dataset(tmp_dir, skip=2L)
#> ! Invalid: Error creating dataset. Could not read schema from
#> '/tmp/RtmpPRG1yT/arrow_test/test.csv'. Is this a 'csv' file?:
#> Could not open CSV input source '/tmp/RtmpPRG1yT/arrow_test/test.csv':
#> Invalid: CSV parse error: Row #2: Expected 1 columns,
#> got 2: This one too, even though it has a comma
# These generate the same error:
open_csv_dataset(tmp_dir, skip=3L)
open_dataset(tmp_dir, format="csv", skip=2L)
schem <- schema(
field(name="X", type=int32()),
field(name="Y", type=int32()),
field(name="Z", type=int32())
)
# Works when schema is supplied (now also need to skip header row)
open_csv_dataset(tmp_dir, skip=3L, schema=schem) |> dplyr::collect()
```
## Version info:
R 4.2.2 on Linux
Arrow 12.0.0
(edited to add version info)
### Component(s)
R
Contributor guide
Assessment
This issue has not been assessed yet.