tamnd / tamnd/firepanda

NaN is a missing value in pandas and a value in firepanda

Open
#170 4 comments 0 reactions 0 assignees View on GitHub
Dominant language
Mojo
Stars
1
Forks
0
PR merge metrics
PR metrics pending

Description

Found by the pandas conformance driver in [firepanda-compat #27](https://github.com/tamnd/firepanda-compat/pull/27). Four of the eighteen remaining failures in the `basics` section are this one thing, and it is a decision rather than a bug, which is why it is an issue and not a pull request.

pandas on the numpy backend has no separate missing value for a float column. NaN is it. `Series([1.0, nan]).count()` is 1, and every reduction with the default `skipna=True` steps over that NaN exactly as it steps over a value that was never there. firepanda follows Arrow, where a null is a bit in the validity bitmap and NaN is an ordinary float that happens to compare false against itself. Both models are defensible and only one of them is the pandas API.

What the suite measured, on the committed corpus:

| case | frame | firepanda | pandas |
|---|---|---|---|
| `basics/count` | `float64_no_nulls`, 0 Arrow nulls and 1 NaN | 64 | 63 |
| `basics/count` | `float64_half_null`, 32 Arrow nulls and 1 NaN | 32 | 31 |
| `basics/sum` | `float64_half_null` | nan | -inf |
| `basics/mean` | `float64_half_null` | nan | -inf |

The last two are worth reading twice, because the difference is not that one of them is nan and the other is not. The column holds a negative infinity. pandas skips the NaN, adds the rest, and the infinity survives to be the answer. firepanda adds the NaN in and one NaN anywhere in a column poisons every float reduction over it. A single NaN in a ten million row column turns every sum, mean, min and max over that column into nan, and the value that a user would call the answer is sitting right there in the data.

There is a second half to this, which is what a reduction over nothing returns. pandas gives NaN, since that is its missing value, and firepanda returns a null:

| case | frame | firepanda | pandas |
|---|---|---|---|
| `basics/mean` | `float64_all_null` | null | nan |
| `basics/min` | `float64_all_null` | null | nan |
| `basics/max` | `float64_all_null` | null | nan |
| `basics/mean` | `int64_all_null` | null | nan |

Under the pandas model those two are one question and not two. NaN is missing on the way in and NaN is what missing looks like on the way out.

The cost of matching is real and should be said out loud rather than discovered later. Treating NaN as missing means a validity read per row is no longer enough, and the reduction kernels have to look at each value as well, which is a compare and a mask per lane in the inner loop of the operations this library exists to make fast. It also means `count` cannot be `len - null_count` on a float column, which is currently a subtraction of two numbers the column already knows.

Three ways out, and I do not think the first is one:

1. Leave it and register the divergence. This makes `count` wrong on any float column with a NaN in it, and `count` is not an exotic call.
2. Match pandas in the float reduction kernels. The per lane cost is a `!=` against itself and a select, which vectorizes, and it applies only to float dtypes, so no integer or string path pays anything.
3. Normalize on the way in, so that a NaN read from a file becomes a null and the kernels stay as they are. This is cheapest at run time and it is lossy: a user who put a NaN in on purpose cannot get it back out, and the Arrow round trip stops being a round trip.

My reading is 2. It keeps the data faithful, the cost lands only where the semantics are wanted, and it is the only one of the three where firepanda can claim the pandas answer without an asterisk. But this is a semantics decision with a performance bill attached and it should be argued before it is written.

Contributor guide

Open the contributing guide

Research direction

Start with the pandas conformance driver and firepanda-compat #27, then review the committed corpus cases in the basics section for count, sum, mean, min, and max. Resolve whether firepanda should treat NaN as missing before implementation; done means the selected semantics match the pandas results while accounting for the stated reduction-kernel performance cost.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data-engineering
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.