apache / apache/arrow

[R] integer coercion on parition variables

Open
#33,210 1 comment 0 reactions 0 assignees View on GitHub
Component: R Type: bug
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 18h
Merged PRs (30d)
91

Description

If we partition on a character-string column that is "integer-like", arrow assumes the resulting partition variable is an integer.  This is particularly surprising because in most other cases, making a column a partition variable coerces it to a character (makes sense, path names are strings), but in this case we have the opposite coercion where a string becomes an integer.  Consider this reprex:

```java

library(arrow)f <- tempfile()
df <- data.frame(x = c("00", "06", "18"))
df |> write_dataset(f, partitioning = "x")
open_dataset(f) # x is now an integer
```
```java

## fails
open_dataset(f) |> filter(x == "00")

```
changing this behavior could break existing code I think, so maybe this shouldn't be considered a bug, but I think it merits discussion.

(For instance, we often see large datasets, e.g. from NOAA csv etc, that use zero-padded integers in path names to indicate file-specific metadata.  One side effect of the type-coercion is that it also strips  the zero-padding)

**Reporter**: [Carl Boettiger](https://issues.apache.org/jira/browse/ARROW-18006) / @cboettig

**Note**: *This issue was originally created as [ARROW-18006](https://issues.apache.org/jira/browse/ARROW-18006). Please see the [migration documentation](https://github.com/apache/arrow/issues/14542) for further details.*

Contributor guide

Open the contributing guide

Research direction

Reproduce the behavior from the R reprex using write_dataset(), partitioning = "x", and open_dataset(). Trace how the character partition values "00", "06", and "18" are inferred and how that affects filtering; done requires an agreed behavior that accounts for compatibility and preserves the intended partition values.

Written by the indexing model from the issue text.

Assessment

Tech stack
r
Domain
data-engineering
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.