tamnd / tamnd/firepanda

tolist on a temporal column gives the stored integer rather than a datetime

Open
#348 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Mojo
Stars
1
Forks
0
Avg merge
1h 31m
Merged PRs (30d)
640

Description

Found while writing the tests for #345.

`Series.tolist()` on a timestamp column answers the raw stored count rather than a Python `datetime`, and the same on a duration column answers the raw count rather than a `timedelta`.

table = pa.table({"t": pa.array([datetime.datetime(2026, 1, 1)], type=pa.timestamp("us"))})
column = firepanda.from_arrow(table)["t"]
column.tolist() # [1767225600000000]
pd.Series(...).tolist() # [Timestamp('2026-01-01 00:00:00')]

Reading the same column back through `pyarrow.array(column)` gives the exact `datetime` objects pandas gives, so nothing is wrong with the values or with the type that travels across the C Data Interface. The gap is entirely in `tolist`, which is reading the buffer and building Python integers without asking the logical type what the integers mean.

The test file works around it by reading answers back through Arrow and says so in its module docstring, which is honest and is not something a user should have to do.

## Why it matters more than it looks

`tolist` is the ordinary way a person gets values out of a column and into plain Python, and answering an integer where pandas answers a timestamp is a silent wrong answer rather than a missing feature. Nothing raises. A program that formats the result gets a number that looks like it could be meaningful.

It is also the same question #347 raises about `dt.to_pydatetime`, from the other end. Both are asking what leaving a firepanda column for ordinary Python objects should look like, and they should get the same answer rather than two.

## Scope

- `tolist` on a timestamp column gives `datetime.datetime`, at all four units, with a missing row giving `None` the way it does for every other type
- On a zoned column the objects carry the zone, which is what pandas does
- On a duration column, `datetime.timedelta`
- On a date column, `datetime.date`
- The dispatch reads the logical type rather than the storage dtype, which is the same rule the Arrow export already follows

## Exit

A test comparing `tolist` against pandas over the temporal frames in the corpus, at every unit, with a missing row in each, and the workaround comment in `python/tests/test_dt.py` deleted.

Contributor guide

Open the contributing guide

Research direction

Start at the Series.tolist entry point and read python/tests/test_dt.py, including its current Arrow-based workaround. Compare to pandas across the temporal frames in the corpus, covering all timestamp units, missing rows, zoned timestamps, durations, and dates; done means the workaround comment is removed and the results match pandas.

Written by the indexing model from the issue text.

Assessment

Tech stack
pandas, python
Domain
api, data
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
68/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.