apache / apache/arrow

[R] Supper encoding options for CSVs in open_dataset

Open
#31,415 3 comments 0 reactions 0 assignees View on GitHub
Component: R Type: task
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 18h
Merged PRs (30d)
91

Description

The encoding options are passed when a single file is read with read_delim_arrow, but not when opening a folder with open_dataset.

read_delim_arrow creates a reader using CsvTableReader$create (which is what is tested in the package's tests).

open_dataset creates a factory and I'm unable to follow what happens when $Finish() is called.

 

Also, the documentation ("CsvReadOptions" page) lists the "encoding" option under "CsvConvertOptions$create()" instead of "CsvReadOptions$create()"

 
```r

library(dplyr)
library(arrow)
# Opens one file just fine:
one_file <- arrow::read_delim_arrow(
  "test/Test1.txt", 
  as_data_frame = FALSE,
  delim = ";",
  read_options = CsvReadOptions$create(encoding = "ISO-8859-1")
)
collect(one_file)
 
# Can't open the folder that has "Test1.txt" properly, results in Column2 being typed as binary
one_folder <- arrow::open_dataset(
  "test", 
  delim = ";",
  read_options = CsvReadOptions$create(encoding = "ISO-8859-1")
)
collect(one_folder)
 
# Even when specify the schema
one_folder_w_schema <- arrow::open_dataset(
  "test", 
  schema = Schema$create(Column1 = string(), Column2 = string()),
  format = FileFormat$create("text", skip_rows = 1L, delimiter = ";", column_names = c("Column1", "Column2"),
                             read_options = CsvReadOptions$create(encoding = "ISO-8859-1"))
  
)
collect(one_folder_w_schema)
```
 

**Reporter**: [Gregoire Leleu](https://issues.apache.org/jira/browse/ARROW-15992)
#### Related issues:
- [[C++][Dataset] Support Latin-1 encoding](https://github.com/apache/arrow/issues/31423) (is blocked by)
#### Original Issue Attachments:
- [Test1.txt](https://issues.apache.org/jira/secure/attachment/13041397/Test1.txt)

**Note**: *This issue was originally created as [ARROW-15992](https://issues.apache.org/jira/browse/ARROW-15992). Please see the [migration documentation](https://github.com/apache/arrow/issues/14542) for further details.*

Contributor guide

Open the contributing guide

Research direction

Start with the R entry points read_delim_arrow and open_dataset, then compare CsvTableReader$create with the factory path reached by $Finish(). Review the package tests for CsvTableReader$create and the CsvReadOptions documentation. Done means the folder example honors the encoding and the option is listed under CsvReadOptions$create().

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp, r
Domain
data-engineering
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.