apache / apache/arrow

[Python] Use saved pandas metadata to determine default timestamp_as_object in to_pandas()

Open
#33,365 1 comment 0 reactions 0 assignees View on GitHub
Component: Python Type: enhancement
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 18h
Merged PRs (30d)
91

Description

If there is pandas metadata present and it indicates object dtype, we should use that to avoid conversion to `datetime64[ns]`, but keep `datetime` objects in `to_pandas`.

**Reporter**: [Alenka Frim](https://issues.apache.org/jira/browse/ARROW-18175) / @AlenkaF
#### Related issues:
- [[Python] Year 2263 or later datetimes get mangled when written using pandas](https://github.com/apache/arrow/issues/24959) (is related to)

**Note**: *This issue was originally created as [ARROW-18175](https://issues.apache.org/jira/browse/ARROW-18175). Please see the [migration documentation](https://github.com/apache/arrow/issues/14542) for further details.*

Contributor guide

Open the contributing guide

Research direction

Begin at the Python to_pandas() entry point and trace how pandas metadata is read when timestamp_as_object is chosen. Check the related datetime issue for context, then identify the relevant conversion tests. Done means metadata indicating object dtype preserves datetime objects instead of converting them to datetime64[ns].

Written by the indexing model from the issue text.

Assessment

Tech stack
pandas, python
Domain
data
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.