apache / apache/arrow

[R] Using props$set_coerce_int96_timestamp_unit in open_dataset()

Open
#34,062 6 comments 0 reactions 0 assignees View on GitHub
Component: R Type: usage
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 13h
Merged PRs (30d)
88

Description

### Describe the usage question you have. Please include as many useful details as possible.

Hello there,

When reading a parquet file using `read_parquet()` that contains int96 dates, I do this:

```
props <- arrow::ParquetArrowReaderProperties$create()

props$set_coerce_int96_timestamp_unit(arrow::TimeUnit$MILLI)
```

Then I pass props to the read_parquet function to read the correct date format.

Example here:
```
tf <- tempfile()

table <- Table$create(some_datetime = as.POSIXct("9999-12-31 00:00:00.000"))
write_parquet(table, tf, use_deprecated_int96_timestamps = TRUE)
props <- ParquetArrowReaderProperties$create()
props$set_coerce_int96_timestamp_unit(TimeUnit$MILLI)

result <- read_parquet(
tf,
as_data_frame = TRUE,
props = props
)
result
```

When I'm working with arrow datasets, passing the props using the ... in the `open_dataset()` function, I get
```
Error in ParquetFragmentScanOptions$create(...) :
unused argument (props = )
```
I went through the repo and couldn't find the `set_coerce_int96_timestamp_unit` as an option in `ParquetFragmentScanOptions$create(...)`.

How do I read parquet files using open_dataset() using set_coerce_int96_timestamp_unit?

Thank you
Mike

### Component(s)

R

Contributor guide

Open the contributing guide

Research direction

Start at the R open_dataset() entry point and ParquetFragmentScanOptions$create(...), then compare how read_parquet() accepts ParquetArrowReaderProperties. Determine how set_coerce_int96_timestamp_unit can be applied through open_dataset(); done means parquet datasets honor that setting, with coverage for the supplied int96 timestamp example.

Written by the indexing model from the issue text.

Assessment

Tech stack
r
Domain
data-engineering
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.