apache / apache/arrow

Preserve unit from Pandas timestamp when creating arrow scalar

Open
#36,035 1 comment 0 reactions 0 assignees View on GitHub
Component: Python Type: enhancement
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 13h
Merged PRs (30d)
88

Description

### Describe the enhancement requested

```
>>> ts = pd.Timestamp("2020-07-28 21:10:53.114429867")
>>> ts.unit
'ns'
>>> pa_s = pa.scalar(ts)
>>> pa_s.type
TimestampType(timestamp[us])
>>>
```

Since the pandas unit was nanoseconds, would be nice if that were preserved in the inferred type, without having to specify ns explicitly, as microseconds results in precision loss.

### Component(s)

Python

Contributor guide

Open the contributing guide

Research direction

Reproduce the issue with the shown pa.scalar(ts) example and inspect the Python scalar conversion path used for pandas.Timestamp values. Confirm the inferred timestamp type preserves the pandas nanosecond unit and avoids precision loss, then add or update coverage for this example.

Written by the indexing model from the issue text.

Assessment

Tech stack
pandas, python
Domain
data
Issue type
Feature
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.