apache / apache/arrow

[R] Add option to attempt 32-bit integer type inference in CSV reader

Open
#30,083 1 comment 0 reactions 0 assignees View on GitHub
Component: R Type: task
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 13h
Merged PRs (30d)
88

Description

Expose ARROW-13028 to the R bindings so that users can choose to have `read_csv_arrow(as_data_frame = FALSE)` and `open_dataset(format = "csv)` infer 32-bit integer fields instead of always inferring 64-bit integer fields for all integers (the current behavior).

Note that there is an existing option in the R bindings that controls something similar: `arrow.int64_downcast`. See ARROW-10093 for details. I think we can _not_ reuse this option to control the CSV reader, behavior because (a) users might want to control these behaviors separately, and (b) the default value of `arrow.int64_downcast` is `TRUE` which does not align with the existing behavior of the CSV reader (always inferring 64-bit integer fields, i.e. _not_ downcasting) and we probably want to retain that as the default behavior. So we will want to add a new argument or a new option to control this.

**Reporter**: [Ian Cook](https://issues.apache.org/jira/browse/ARROW-14528) / @ianmcook
#### Related issues:
- [[C++] unify_schemas can't handle int64 + double, affects CSV dataset](https://github.com/apache/arrow/issues/30245) (is related to)
- [[C++] CSV add convert option to attempt 32bit number inferences](https://github.com/apache/arrow/issues/28740) (depends upon)

**Note**: *This issue was originally created as [ARROW-14528](https://issues.apache.org/jira/browse/ARROW-14528). Please see the [migration documentation](https://github.com/apache/arrow/issues/14542) for further details.*

Contributor guide

Open the contributing guide

Research direction

Start with the R bindings for read_csv_arrow(as_data_frame = FALSE) and open_dataset(format = "csv"), then review ARROW-13028 and the related C++ issue ARROW-28740. Define a separate option while preserving the current 64-bit default and the existing arrow.int64_downcast behavior; done means both R entry points can request 32-bit integer inference.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp, r
Domain
data
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.