A category column can only hold text, so astype("category") on numbers refuses
- Dominant language
- Mojo
- Stars
- 1
- Forks
- 0
- PR merge metrics
- PR metrics pending
Description
`pd.Series([1, 2, 1]).astype("category").cat.categories` is `Index([1, 2], dtype='int64')`. The same call in firepanda raises `cast: a column of int64 cannot be encoded as a category, and categories are text`.
The categories of a firepanda dictionary column are a `StringArray`, which is a field on `AnyArray` of exactly that type, so a category list of numbers has nowhere to live. Everything downstream follows from that: `encode_dictionary` takes a `StringArray`, `decode_dictionary` returns one, and the seven `cat_*` members on `Series` are all written against text.
Casting to text first and encoding that is not the same operation. pandas keeps the categories at the dtype the column had, so a caller who encodes an int64 column and reads `cat.categories` back gets int64 values and can compare them to numbers. Handing back strings would give the right number of categories, the right codes and the wrong type, which is the kind of near miss that is worse than a refusal.
Arrow allows any type as a dictionary's value type, so this is a firepanda limitation rather than a format one, and a pandas categorical of numbers imported over the C data interface is a shape firepanda cannot currently represent at all.
Measured by the conformance suite: `categorical/astype-category` runs on three frames and firepanda answers two of them. The third is `keys_awkward`, whose last column is int64, and that run is the one failure left in the `categorical` section.
The work is making the categories a column rather than a `StringArray`, which touches the dictionary constructor, the encoder, the decoder, the accessor and the Arrow reader and writer. Worth doing as its own piece rather than folded into something else.
Milestone: [M6](https://github.com/tamnd/firepanda/issues/8).
Contributor guide
Research direction
Start with the categorical/astype-category conformance case, then trace the dictionary constructor, encode_dictionary, decode_dictionary, cat_* accessor, and Arrow reader and writer mentioned in the issue. Preserve numeric category values and make the three-frame conformance run pass, including keys_awkward, without converting categories to text.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- pandas
- Domain
- data-engineering
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100