tamnd / tamnd/firepanda

A reduction over an integer column with nulls: which pandas are we a copy of

Open
#171 3 comments 0 reactions 0 assignees View on GitHub
Dominant language
Mojo
Stars
1
Forks
0
PR merge metrics
PR metrics pending

Description

The other half of what the pandas conformance driver found in [firepanda-compat #27](https://github.com/tamnd/firepanda-compat/pull/27). Six of the eighteen remaining `basics` failures are this, and like [#170](https://github.com/tamnd/firepanda/issues/170) it is a decision rather than a defect.

pandas on the numpy backend has no nullable integer. An int64 column with a null in it does not survive as an int64 column, it arrives as float64 with NaN in the null positions, and every reduction over it is therefore a float reduction. firepanda keeps the column as int64 with a validity bitmap, which is the Arrow answer and the better one, and its reductions return int64.

| case | frame | firepanda | pandas |
|---|---|---|---|
| `basics/sum` | `int64_half_null` | int64 | double |
| `basics/sum` | `int64_all_null` | int64 | double |
| `basics/min` | `int64_half_null` | int64 | double |
| `basics/min` | `int64_all_null` | int64 | double |
| `basics/max` | `int64_half_null` | int64 | double |
| `basics/max` | `int64_all_null` | int64 | double |

The type is not the only thing that differs, and this is the part that makes it more than a cast. pandas sums `int64_half_null` to -1.4757395252454973e+20. No int64 sum can produce that number, because the widening happened before the addition did, so pandas is not returning a wrapped total in a float box, it is returning a total that never wrapped. firepanda's int64 sum of the same column wraps, correctly, because that is what an int64 sum does.

So a cast of the result would agree on the dtype and still disagree on the value. Matching pandas here means widening the input, not the output, which is exactly the shape of the mean fix that went in with the empty chunk work: pandas converts and then reduces, and reducing and then converting is a different operation.

Worth noting where this stops. This is entirely about the numpy backed dtypes. pandas also has `Int64`, the nullable extension dtype, and a column of that type sums to a nullable `Int64` with no widening anywhere, which is what firepanda already does. So one reading of this issue is that firepanda's answer is right and it is answering a question the conformance corpus did not ask, because the corpus goes through `to_pandas()` and lands on the numpy side.

That makes the decision a question about which pandas firepanda is a copy of, and it is bigger than these six cases:

1. The numpy backed dtypes, which is what `pd.read_parquet` and `pd.DataFrame` still give you by default in 3.0 and what almost every existing pandas program is written against.
2. The Arrow backed dtypes, where firepanda's current behaviour is already right and where the memory model matches what firepanda actually is.

Whichever way this goes it belongs in the specification before it goes in the kernels, and the conformance suite needs to know which one it is measuring, because right now it measures the first and firepanda implements the second. If the answer is that firepanda targets the Arrow backed semantics, then these six are divergences with a reason a user would accept and they get registered as such, and the corpus grows a set of frames that arrive as extension dtypes so the suite is comparing like with like. If the answer is the numpy semantics, then the widening moves into the read path and these six become bugs.

I lean towards the second being the target and the first being what conformance is scored against, since the point of the pandas API is that existing programs run, and an existing program was written against the dtype pandas gives it today. But that is a call about what firepanda is for and not one to make inside a kernel.

Contributor guide

Open the contributing guide

Research direction

Start with the conformance driver in firepanda-compat #27 and its basics/sum, min, and max cases, then review how to_pandas() determines the dtype semantics being compared. Done means recording the chosen pandas target in the specification and aligning the conformance corpus so these six reductions compare the intended dtype family.

Written by the indexing model from the issue text.

Assessment

Tech stack
pandas, python
Domain
data
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.