[R] Supper encoding options for CSVs in open_dataset
- Dominant language
- C++
- Stars
- 17.1k
- Forks
- 4.3k
- Avg merge
- 3d 18h
- Merged PRs (30d)
- 91
Description
The encoding options are passed when a single file is read with read_delim_arrow, but not when opening a folder with open_dataset.
read_delim_arrow creates a reader using CsvTableReader$create (which is what is tested in the package's tests).
open_dataset creates a factory and I'm unable to follow what happens when $Finish() is called.
Also, the documentation ("CsvReadOptions" page) lists the "encoding" option under "CsvConvertOptions$create()" instead of "CsvReadOptions$create()"
```r
library(dplyr)
library(arrow)
# Opens one file just fine:
one_file <- arrow::read_delim_arrow(
"test/Test1.txt",
as_data_frame = FALSE,
delim = ";",
read_options = CsvReadOptions$create(encoding = "ISO-8859-1")
)
collect(one_file)
# Can't open the folder that has "Test1.txt" properly, results in Column2 being typed as binary
one_folder <- arrow::open_dataset(
"test",
delim = ";",
read_options = CsvReadOptions$create(encoding = "ISO-8859-1")
)
collect(one_folder)
# Even when specify the schema
one_folder_w_schema <- arrow::open_dataset(
"test",
schema = Schema$create(Column1 = string(), Column2 = string()),
format = FileFormat$create("text", skip_rows = 1L, delimiter = ";", column_names = c("Column1", "Column2"),
read_options = CsvReadOptions$create(encoding = "ISO-8859-1"))
)
collect(one_folder_w_schema)
```
**Reporter**: [Gregoire Leleu](https://issues.apache.org/jira/browse/ARROW-15992)
#### Related issues:
- [[C++][Dataset] Support Latin-1 encoding](https://github.com/apache/arrow/issues/31423) (is blocked by)
#### Original Issue Attachments:
- [Test1.txt](https://issues.apache.org/jira/secure/attachment/13041397/Test1.txt)
**Note**: *This issue was originally created as [ARROW-15992](https://issues.apache.org/jira/browse/ARROW-15992). Please see the [migration documentation](https://github.com/apache/arrow/issues/14542) for further details.*
Contributor guide
Research direction
Start with the R entry points read_delim_arrow and open_dataset, then compare CsvTableReader$create with the factory path reached by $Finish(). Review the package tests for CsvTableReader$create and the CsvReadOptions documentation. Done means the folder example honors the encoding and the option is listed under CsvReadOptions$create().
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp, r
- Domain
- data-engineering
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100