tamnd / tamnd/firepanda

The constructor door: a None and a NaN, after the read path was decided

Open
#301 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Mojo
Stars
1
Forks
0
Avg merge
1h 31m
Merged PRs (30d)
640

Description

[#300](https://github.com/tamnd/firepanda/pull/300) decided where a number goes when there is no number, and turned one of the two doors. This is the other one.

`firepanda.read_csv` now widens a numeric column with a missing row to float64 with a NaN in the gap, because that is what pandas does when it reads a file. `firepanda.DataFrame({"i": [1, None, 3]})` does not. It builds an int64 column with a cleared bit, and `tolist()[1] is None`.

So the same binding now answers the same question two ways depending on which door the data came through, and that will catch somebody out. A user who reads a CSV gets float64, a user who builds the same frame from a literal gets int64, and nothing in either signature says why.

## The argument for the current behaviour

Document 18 section 3 makes it, and it is not a bad argument. A caller who writes `None` and a caller who writes `float("nan")` have said two different things in Python, and the constructor is the one place firepanda can see the difference and keep it. `python/tests/test_construct.py::test_a_nan_is_a_value_and_a_none_is_not` asserts exactly that: `DataFrame({"a": [1.0, float("nan"), None]})` gives back a NaN at index 1 and a `None` at index 2, while `count()` is 1 and `hasnans` is True, so what counts them agrees with pandas even though what stores them does not.

An empty CSV field has no such distinction to preserve. There is one kind of gap in a file, so there is nothing lost by resolving it the pandas way.

## The argument against

pandas answers `float64` for `pd.DataFrame({"i": [1, None, 3]})` and `object` for `pd.DataFrame({"a": [None, None]})`, and the front of this project says `import firepanda as pd` and your program keeps running. A program that branches on `df["i"].dtype` gets a different answer from the two libraries, and the divergence document is not something the program reads.

The document 18 argument was also written before the compatibility goal was measured. It is worth asking whether it survives that, rather than inheriting it.

## What is in scope

- `firepanda.DataFrame` and `firepanda.Series` built from Python sequences.
- `firepanda.Index`, which has the same shape and its own written divergence in `python/tests/test_index.py::test_a_missing_label_keeps_its_dtype`.
- Whatever `docs/specs/18-making-a-frame-out-of-python.md` section 3 and `docs/specs/20-editing-an-index.md` have to say afterwards, since one of them has to change whichever way this goes.

## What is not

`firepanda.from_arrow`. That door keeps Arrow's answer and #300 says why at length: the C Data Interface promises a consumer that what came in comes out, and it is how firepanda talks to Polars and DuckDB as well as to pandas.

Refs [#171](https://github.com/tamnd/firepanda/issues/171), and `docs/specs/20-the-missing-number.md` in tamnd/firepanda-compat.

Contributor guide

Open the contributing guide

Research direction

Start with python/tests/test_construct.py::test_a_nan_is_a_value_and_a_none_is_not and python/tests/test_index.py::test_a_missing_label_keeps_its_dtype, then read section 3 of docs/specs/18-making-a-frame-out-of-python.md and docs/specs/20-editing-an-index.md. Compare the constructor and Index behavior with the stated pandas compatibility goal, and update the affected tests and specifications once a consistent decision is reached; leave firepanda.from_arrow outside the change.

Written by the indexing model from the issue text.

Assessment

Tech stack
pandas, python
Domain
api, data
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.