[R] integer coercion on parition variables
- Dominant language
- C++
- Stars
- 17.1k
- Forks
- 4.3k
- Avg merge
- 3d 18h
- Merged PRs (30d)
- 91
Description
If we partition on a character-string column that is "integer-like", arrow assumes the resulting partition variable is an integer. This is particularly surprising because in most other cases, making a column a partition variable coerces it to a character (makes sense, path names are strings), but in this case we have the opposite coercion where a string becomes an integer. Consider this reprex:
```java
library(arrow)f <- tempfile()
df <- data.frame(x = c("00", "06", "18"))
df |> write_dataset(f, partitioning = "x")
open_dataset(f) # x is now an integer
```
```java
## fails
open_dataset(f) |> filter(x == "00")
```
changing this behavior could break existing code I think, so maybe this shouldn't be considered a bug, but I think it merits discussion.
(For instance, we often see large datasets, e.g. from NOAA csv etc, that use zero-padded integers in path names to indicate file-specific metadata. One side effect of the type-coercion is that it also strips the zero-padding)
**Reporter**: [Carl Boettiger](https://issues.apache.org/jira/browse/ARROW-18006) / @cboettig
**Note**: *This issue was originally created as [ARROW-18006](https://issues.apache.org/jira/browse/ARROW-18006). Please see the [migration documentation](https://github.com/apache/arrow/issues/14542) for further details.*
Contributor guide
Research direction
Reproduce the behavior from the R reprex using write_dataset(), partitioning = "x", and open_dataset(). Trace how the character partition values "00", "06", and "18" are inferred and how that affects filtering; done requires an agreed behavior that accounts for compatibility and preserves the intended partition values.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- r
- Domain
- data-engineering
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100