tamnd / tamnd/firepanda

M. The dt accessor, 42 names on a column that can be read and not asked

Open
#287 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Mojo
Stars
1
Forks
0
PR merge metrics
PR metrics pending

Description

The fifth and last item of #270. Specification: [`14-the-dt-accessor.md`](https://github.com/tamnd/firepanda-compat/blob/main/docs/specs/14-the-dt-accessor.md).

A datetime column exists now and nothing can ask it anything. The reader opens a timestamp at any unit with or without a zone (#272), a date (#272) and a duration (#275), the writer writes all three back, and the conformance suite's `temporal` section has no failures left in it. It also has no passes. All 122 of its runs report `unimplemented`, because a column that can be read and not asked is a column with no methods on it.

### The measurement

From the run on 2026-09-07, against pandas 3.0.3 and firepanda 0.6.51. The `temporal` section is 122 runs over 80 distinct cases, all `unimplemented`.

| group | cases | runs |
| --- | --- | --- |
| the `dt` accessor and what holds it up | 68 | 108 |
| resampling, `asfreq`, DST arithmetic, the ambiguous time policy | 12 | 14 |

108 of 122 are this issue and 14 are M7 (#9).

### The size of it

`dt` is 42 names and 13 callables, because 29 of them are properties. 19 parameters hang off the 13.

The 29 properties are `date`, `day`, `day_of_week`, `day_of_year`, `dayofweek`, `dayofyear`, `days_in_month`, `daysinmonth`, `freq`, `hour`, `is_leap_year`, `is_month_end`, `is_month_start`, `is_quarter_end`, `is_quarter_start`, `is_year_end`, `is_year_start`, `microsecond`, `minute`, `month`, `nanosecond`, `quarter`, `second`, `time`, `timetz`, `tz`, `unit`, `weekday` and `year`.

The 13 callables are `as_unit`, `ceil`, `day_name`, `floor`, `isocalendar`, `month_name`, `normalize`, `round`, `strftime`, `to_period`, `to_pydatetime`, `tz_convert` and `tz_localize`.

Six of the 42 are three numbers under six names: `day_of_week`, `dayofweek` and `weekday` are one, `days_in_month` and `daysinmonth` are another. All six exist and three are aliases at the binding layer.

An L3 rate over `dt` measures 13 names while the work is on 42, so the runs are the honest number here.

### Scope

- [ ] **The field extractors.** The 15 calendar properties, the 5 clock properties, and `dt.date`, `dt.time` and `dt.normalize`, which are the same arithmetic returning a column. One civil calendar conversion from a day number to a year, month and day, written once as a kernel over the column, and the fifteen names are cheap after it. Hinnant's `civil_from_days` is the conversion: branch free integer arithmetic over a four hundred year cycle, no table and no loop.
- [x] **The rounding trio.** `dt.floor`, `dt.ceil` and `dt.round` at fixed frequencies, which is arithmetic on the count with no calendar in it. The frequency string parser is shared with M7 and gets written here. `dt.round` rounds a half to even, which is the one behaviour in this group that a reimplementation gets wrong by default. Anything month based is an offset and offsets are M7. Landed in [#290](https://github.com/tamnd/firepanda/pull/290) with `dt.as_unit`, which has no frequency in it and belongs with them because it is the other arithmetic on the count: 9 runs, all passing, and the grammar is written up in [`15-the-frequency-string.md`](https://github.com/tamnd/firepanda-compat/blob/main/docs/specs/15-the-frequency-string.md).
- [x] **The names and the formats.** `dt.day_name`, `dt.month_name`, `dt.strftime` and `dt.isocalendar`. English names, with any other locale refused by name rather than the argument silently ignored. `strftime` is the one item here that is a parser rather than arithmetic, and `isocalendar` is the only member of `dt` that returns a frame. Landed in [#291](https://github.com/tamnd/firepanda/pull/291): 5 runs, all passing, and the grammar is written up in [`16-the-format-string.md`](https://github.com/tamnd/firepanda-compat/blob/main/docs/specs/16-the-format-string.md). The ISO calendar needs no week count table, because the ISO year is the calendar year of the week's Thursday and the ISO week is that Thursday's day of the year over seven. The format string was measured rather than reasoned about, and a padding flag turns out to move a width for exactly thirteen directives and to be ignored by every other one including `%Y`, which is always four digits whatever is in front of it.
- [x] **The zone pair.** `dt.tz`, `dt.tz_localize` and `dt.tz_convert`, for fixed offset conversions and `tz_localize(None)`. The three ambiguous cases, the two nonexistent cases and Lord Howe are M7 and the corpus separates them deliberately. Landed in #345 for the binding, on top of the kernel work that was already there. `tz_localize(None)` crosses the boundary as its own word rather than as an absent argument, because naming a zone keeps the readings and changes what they mean while passing `None` keeps the instants and drops what they were read against, and those are two operations pandas spells as one.
- [ ] **`pandas.to_datetime`**, both the ISO 8601 form and the explicit `format` form. The parser is the one the CSV reader already has, reached from a different door.
- [ ] **`pandas.date_range`** in its fixed frequency form. The offset frequencies are M7.
- [x] **`Series.dtype` on a datetime column.** Eight runs and the cheapest eight in the section, since the type already carries the unit and the zone and `LogicalType.write_to` already spells `datetime64[ns, Europe/Paris]`.
- [x] **The duration column's ten runs.** `duration-dtype`, `duration-days`, `duration-sum`, `duration-mean`, `duration-abs`, `total-seconds`, `timedelta-construct`, `to-timedelta`, `timestamp-minus-timestamp` and `timestamp-plus-duration`. Here rather than in an issue of their own, because subtracting two timestamps produces a duration. Landed in [#293](https://github.com/tamnd/firepanda/pull/293) with the driver entries in [firepanda-compat#62](https://github.com/tamnd/firepanda-compat/pull/62): 10 runs, all passing, and the rules are written up in [`17-the-elapsed-time.md`](https://github.com/tamnd/firepanda-compat/blob/main/docs/specs/17-the-elapsed-time.md). The unit rule turned out to be the smaller half. Promotion is the wrong shape for time, because reconciling two resolutions is a multiplication of every value rather than a widening cast and because a timestamp plus a duration has no common operand type at all, so the answer type comes from a separate function keyed on the operator. A `Timedelta` constant carries its resolution in the spelling and not in the amount, so `Timedelta(90, unit='s')` and `Timedelta(hours=1)` give differently typed answers on the same column. The duration mean goes through float64 and truncates toward zero at both signs, which is reproduced rather than improved.
- [ ] **The driver entries in firepanda-compat** for all 68 cases, landed with the library change rather than after it.

### The thing to test first

A timestamp before 1970. The last second of 1969 lands in a different day from the first second of 1970 only if the division down to days floors rather than truncating toward zero. Mojo's `//` floors, which is right, and the fast paths a kernel reaches for do not: a hardware divide truncates and an arithmetic shift floors only for a power of two divisor, which 86400 is not. So this is correct today and stays correct only if a test fails when it stops flooring.

A field the unit cannot express answers zero rather than raising. A second column has no nanosecond, and `nanosecond-s`, `nanosecond-ms`, `nanosecond-us` and `nanosecond-ns` sit side by side in the corpus for that reason.

### The driver is half the work

#282 is the demonstration. List and struct columns read, and all 36 `nested` runs moved from `fail` to `unimplemented` reporting `Absent: the firepanda driver has no entry for nested/list-len`, which is a message about `drivers/firepanda/main.mojo` and not about this library. A pull request here with no driver entry behind it moves nothing on the board.

### Exit criteria

- [ ] All 29 properties and all 13 callables exist, with all three `weekday` spellings and both `daysinmonth` spellings
- [ ] Every field is correct before 1970 and at all four units
- [ ] The `temporal` section reports passes rather than `unimplemented` for the 68 in scope cases, measured rather than predicted, with the run posted
- [ ] The 14 M7 runs still report `unimplemented` and not a wrong answer
- [ ] #270 is closed by this

### What stays out

`offsets`, `Resampler`, `resample`, `asfreq`, `merge_asof`, time based rolling and the ambiguous and nonexistent local time policy, all of which are #9. `DatetimeIndex` and its 91 callables go with #154 and #155. `Period` and `Interval` are column types of their own and neither is in the corpus, so `dt.to_period` raises by name here.

### Depends on

#272 and #275, both landed.

### Blocks

The temporal half of #9.

Contributor guide

Open the contributing guide

Research direction

Read docs/specs/14-the-dt-accessor.md and inspect the temporal conformance section, starting with the remaining unchecked field extractors, pandas.to_datetime, date_range, and driver entries. Use the existing CSV datetime parser and drivers/firepanda/main.mojo as entry points. Done means the in-scope cases report passes at all four units, including pre-1970 values, with the M7 cases still unimplemented.

Written by the indexing model from the issue text.

Assessment

Tech stack
pandas
Domain
data-engineering
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.