apache / apache/arrow

[R][C++][Dataset] open_dataset and open_csv_dataset do not use skip argument

Open
#35,756 4 comments 0 reactions 1 assignee Claimed by @thisisnic View on GitHub
Component: C++ Component: R Priority: Critical Type: bug
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 13h
Merged PRs (30d)
88

Description

### Describe the bug, including details regarding any error messages, version, and platform.

I noticed that a CSV with header rows can be successfully read with `read_csv_arrow(..., skip=N)`, but not with `open_csv_dataset` unless a schema is provided. An example is below.

I expected `open_csv_dataset` to use the skip argument the same way `read_csv_arrow` does, skipping a fixed number of rows from every CSV, but it seems to not be skipping any. I think this is likely a bug -- maybe in the schema parsing?

```R
library(arrow)

lines <- c(
"This line should be skipped",
"This one too, even though it has a comma",
"X,Y,Z",
"1,2,3",
"4,5,6"
)
tmp_dir <- file.path(tempdir(), "arrow_test")
dir.create(tmp_dir, recursive=TRUE)
tmp_csv <- file.path(tmp_dir, "test.csv")
writeLines(lines, tmp_csv)

# This works as expected, skipping the first two lines and reading headers from the third
read_csv_arrow(tmp_csv, skip=2L)

open_csv_dataset(tmp_dir, skip=2L)
#> ! Invalid: Error creating dataset. Could not read schema from
#> '/tmp/RtmpPRG1yT/arrow_test/test.csv'. Is this a 'csv' file?:
#> Could not open CSV input source '/tmp/RtmpPRG1yT/arrow_test/test.csv':
#> Invalid: CSV parse error: Row #2: Expected 1 columns,
#> got 2: This one too, even though it has a comma

# These generate the same error:
open_csv_dataset(tmp_dir, skip=3L)
open_dataset(tmp_dir, format="csv", skip=2L)

schem <- schema(
field(name="X", type=int32()),
field(name="Y", type=int32()),
field(name="Z", type=int32())
)

# Works when schema is supplied (now also need to skip header row)
open_csv_dataset(tmp_dir, skip=3L, schema=schem) |> dplyr::collect()

```

## Version info:
R 4.2.2 on Linux
Arrow 12.0.0
(edited to add version info)

### Component(s)

R

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.