[R] Using props$set_coerce_int96_timestamp_unit in open_dataset()
- Dominant language
- C++
- Stars
- 17.1k
- Forks
- 4.3k
- Avg merge
- 3d 13h
- Merged PRs (30d)
- 88
Description
### Describe the usage question you have. Please include as many useful details as possible.
Hello there,
When reading a parquet file using `read_parquet()` that contains int96 dates, I do this:
```
props <- arrow::ParquetArrowReaderProperties$create()
props$set_coerce_int96_timestamp_unit(arrow::TimeUnit$MILLI)
```
Then I pass props to the read_parquet function to read the correct date format.
Example here:
```
tf <- tempfile()
table <- Table$create(some_datetime = as.POSIXct("9999-12-31 00:00:00.000"))
write_parquet(table, tf, use_deprecated_int96_timestamps = TRUE)
props <- ParquetArrowReaderProperties$create()
props$set_coerce_int96_timestamp_unit(TimeUnit$MILLI)
result <- read_parquet(
tf,
as_data_frame = TRUE,
props = props
)
result
```
When I'm working with arrow datasets, passing the props using the ... in the `open_dataset()` function, I get
```
Error in ParquetFragmentScanOptions$create(...) :
unused argument (props = )
```
I went through the repo and couldn't find the `set_coerce_int96_timestamp_unit` as an option in `ParquetFragmentScanOptions$create(...)`.
How do I read parquet files using open_dataset() using set_coerce_int96_timestamp_unit?
Thank you
Mike
### Component(s)
R
Contributor guide
Research direction
Start at the R open_dataset() entry point and ParquetFragmentScanOptions$create(...), then compare how read_parquet() accepts ParquetArrowReaderProperties. Determine how set_coerce_int96_timestamp_unit can be applied through open_dataset(); done means parquet datasets honor that setting, with coverage for the supplied int96 timestamp example.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- r
- Domain
- data-engineering
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100