tamnd / tamnd/firepanda

A Python scalar widens the column it is added to, where pandas leaves it alone

Open
#255 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Mojo
Stars
1
Forks
0
PR merge metrics
PR metrics pending

Description

Measured against pandas 3.0.5, on columns brought in through `from_arrow` because that is the only way to get a column that is not int64 from Python today.

| column | `s + s` here | `s + 1` here | `s + 1` in pandas |
| --- | --- | --- | --- |
| int8 | int8 | int64 | int8 |
| int16 | int16 | int64 | int16 |
| int32 | int32 | int64 | int32 |
| uint8 | uint8 | int64 | uint8 |
| uint16 | uint16 | int64 | uint16 |
| uint32 | uint32 | int64 | uint32 |
| float32 | float32 | float64 | float32 |

Column against column is already right and preserves the width exactly. It is only the constant that widens, and it widens every time: an int8 column plus the literal 1 comes back int64, which is eight times the memory for a column that did not need a single bit more than it had.

### Where it comes from

`firepanda/py/ops.mojo` reads a Python `int` as `Value(Int64(...))` and a Python `float` as `Value(Float64(...))`, because those are the widest of each kind and reading a literal into anything narrower would refuse values that fit in a Python int. The kernel then does exactly the right thing with what it was handed: `promote(int8, int64)` is int64, and a typed language cannot answer anything else when one operand really is an int64.

So neither half is wrong on its own. What is lost between them is that a Python `int` is not an int64. It is an untyped literal, and the pandas rule, which is numpy's rule, is that an untyped scalar does not upcast the array it meets. That fact exists at the boundary and there is nowhere to put it.

### Why the binding cannot fix it alone

For a series it could: the binding has the column in its hand and could build the `Value` in the column's own dtype. For a frame it cannot, because `df + 1` meets one column of each dtype the frame has and there is no single type to build the constant in. pandas preserves each column's own dtype there, so the decision has to be made per column, which means it has to be made inside the kernel.

That points at a weak or untyped scalar in `Value`, one that carries a number and the fact that nobody said what type it was, and a `promote` that takes the column's type when it meets one. The value still has to be checked against the column's range, and what happens when it does not fit is the part worth measuring against pandas before writing anything: numpy has changed its mind about this more than once.

### Why it matters twice

It is a conformance failure, which the suite sees as `dtype int64, expected int8` on the `integer_edges` frame. It is also a resource regression of the plainest kind, because the smallest arithmetic anybody writes is `df["x"] + 1` and the answer to it is currently eight times larger than the question.

Found while measuring the Python arithmetic surface against the conformance suite.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.