apache / apache/arrow

[R] Microsecond is not sufficient unit for POSIXct

Open
#32,693 4 comments 0 reactions 0 assignees View on GitHub
Component: R Type: enhancement
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 18h
Merged PRs (30d)
91

Description

I believe the `POSIXct` type or R currently corresponds to the Arrow `timestamp[us, tz=UTC]` type.

```r

lubridate::as_datetime(0) |> arrow::infer_type()
#> Timestamp
#> timestamp[us, tz=UTC]
```

```r

lubridate::as_datetime("1970-01-01 00:00:00.0000001") |>
arrow::arrow_table(x = _)
#> Table
#> 1 rows x 1 columns
#> $x
```

```r

df_a <- lubridate::as_datetime("1970-01-01 00:00:00.0000001") |>
arrow::arrow_table(x = _) |>
as.data.frame()

df_b <- lubridate::as_datetime("1970-01-01 00:00:00.0000001") |>
tibble::tibble(x = _)

waldo::compare(df_a, df_b)
#> `old$x`: "1970-01-01"
#> `new$x`: "1970-01-01 00:00:00"
```

However, as shown below, POSIXct may hold data finer than a microsecond.

```r

lubridate::as_datetime(0.000000001) |> as.numeric()
#> [1] 1e-09
lubridate::as_datetime("1970-01-01 00:00:00.0000001") |> as.numeric()
#> [1] 1.192093e-07
```

I don't know why it is currently set in microseconds, but is there any reason not to set it in nanoseconds?

**Reporter**: [SHIMA Tatsuya](https://issues.apache.org/jira/browse/ARROW-17424) / @eitsupi
#### Related issues:
- [[R] Convert a column as a sub-second timestamp from CSV file with the `T` col type option](https://github.com/apache/arrow/issues/31061) (relates to)
- [[R] `lubridate::as_datetime()` in dplyr query should be able to handle time in sub seconds](https://github.com/apache/arrow/issues/32694) (is related to)

**Note**: *This issue was originally created as [ARROW-17424](https://issues.apache.org/jira/browse/ARROW-17424). Please see the [migration documentation](https://github.com/apache/arrow/issues/14542) for further details.*

Contributor guide

Open the contributing guide

Research direction

Start with the lubridate::as_datetime(), arrow::infer_type(), and arrow_table() examples in the issue, then trace how R POSIXct values are converted to Arrow timestamps. Compare values with sub-microsecond precision and confirm that the chosen timestamp unit preserves them without changing ordinary datetime behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
r
Domain
data
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.