tolist on a temporal column gives the stored integer rather than a datetime
- Dominant language
- Mojo
- Stars
- 1
- Forks
- 0
- Avg merge
- 1h 31m
- Merged PRs (30d)
- 640
Description
Found while writing the tests for #345.
`Series.tolist()` on a timestamp column answers the raw stored count rather than a Python `datetime`, and the same on a duration column answers the raw count rather than a `timedelta`.
table = pa.table({"t": pa.array([datetime.datetime(2026, 1, 1)], type=pa.timestamp("us"))})
column = firepanda.from_arrow(table)["t"]
column.tolist() # [1767225600000000]
pd.Series(...).tolist() # [Timestamp('2026-01-01 00:00:00')]
Reading the same column back through `pyarrow.array(column)` gives the exact `datetime` objects pandas gives, so nothing is wrong with the values or with the type that travels across the C Data Interface. The gap is entirely in `tolist`, which is reading the buffer and building Python integers without asking the logical type what the integers mean.
The test file works around it by reading answers back through Arrow and says so in its module docstring, which is honest and is not something a user should have to do.
## Why it matters more than it looks
`tolist` is the ordinary way a person gets values out of a column and into plain Python, and answering an integer where pandas answers a timestamp is a silent wrong answer rather than a missing feature. Nothing raises. A program that formats the result gets a number that looks like it could be meaningful.
It is also the same question #347 raises about `dt.to_pydatetime`, from the other end. Both are asking what leaving a firepanda column for ordinary Python objects should look like, and they should get the same answer rather than two.
## Scope
- `tolist` on a timestamp column gives `datetime.datetime`, at all four units, with a missing row giving `None` the way it does for every other type
- On a zoned column the objects carry the zone, which is what pandas does
- On a duration column, `datetime.timedelta`
- On a date column, `datetime.date`
- The dispatch reads the logical type rather than the storage dtype, which is the same rule the Arrow export already follows
## Exit
A test comparing `tolist` against pandas over the temporal frames in the corpus, at every unit, with a missing row in each, and the workaround comment in `python/tests/test_dt.py` deleted.
Contributor guide
Research direction
Start at the Series.tolist entry point and read python/tests/test_dt.py, including its current Arrow-based workaround. Compare to pandas across the temporal frames in the corpus, covering all timestamp units, missing rows, zoned timestamps, durations, and dates; done means the workaround comment is removed and the results match pandas.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- pandas, python
- Domain
- api, data
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 68/100