tamnd / tamnd/firepanda

A category column can only hold text, so astype("category") on numbers refuses

Open
#450 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Mojo
Stars
1
Forks
0
PR merge metrics
PR metrics pending

Description

`pd.Series([1, 2, 1]).astype("category").cat.categories` is `Index([1, 2], dtype='int64')`. The same call in firepanda raises `cast: a column of int64 cannot be encoded as a category, and categories are text`.

The categories of a firepanda dictionary column are a `StringArray`, which is a field on `AnyArray` of exactly that type, so a category list of numbers has nowhere to live. Everything downstream follows from that: `encode_dictionary` takes a `StringArray`, `decode_dictionary` returns one, and the seven `cat_*` members on `Series` are all written against text.

Casting to text first and encoding that is not the same operation. pandas keeps the categories at the dtype the column had, so a caller who encodes an int64 column and reads `cat.categories` back gets int64 values and can compare them to numbers. Handing back strings would give the right number of categories, the right codes and the wrong type, which is the kind of near miss that is worse than a refusal.

Arrow allows any type as a dictionary's value type, so this is a firepanda limitation rather than a format one, and a pandas categorical of numbers imported over the C data interface is a shape firepanda cannot currently represent at all.

Measured by the conformance suite: `categorical/astype-category` runs on three frames and firepanda answers two of them. The third is `keys_awkward`, whose last column is int64, and that run is the one failure left in the `categorical` section.

The work is making the categories a column rather than a `StringArray`, which touches the dictionary constructor, the encoder, the decoder, the accessor and the Arrow reader and writer. Worth doing as its own piece rather than folded into something else.

Milestone: [M6](https://github.com/tamnd/firepanda/issues/8).

Contributor guide

Open the contributing guide

Research direction

Start with the categorical/astype-category conformance case, then trace the dictionary constructor, encode_dictionary, decode_dictionary, cat_* accessor, and Arrow reader and writer mentioned in the issue. Preserve numeric category values and make the three-frame conformance run pass, including keys_awkward, without converting categories to text.

Written by the indexing model from the issue text.

Assessment

Tech stack
pandas
Domain
data-engineering
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.