A categorical has no type object, so there is no way to say the categories and the order at once
- Dominant language
- Mojo
- Stars
- 1
- Forks
- 0
- PR merge metrics
- PR metrics pending
Description
`Series.cat` is here now ([#432](https://github.com/tamnd/firepanda/pull/432) closing [#428](https://github.com/tamnd/firepanda/issues/428)), so a caller can read the categories, read the codes, turn the ordering on and change what the categories are. What is still missing is the type.
## What is missing
`CategoricalDtype` is the one that matters most, because it is how pandas says both halves of a categorical's type at once. A type name is a string and a string cannot carry a list of categories or an ordering, which is why `astype("category")` gives sorted categories and unordered in both libraries. pandas' answer is an object: `astype(CategoricalDtype(["small", "medium", "large"], ordered=True))` builds the column with the categories the caller wants, in the order the caller wants, in one call. Firepanda can reach the same column in three calls now, `astype("category").cat.set_categories(names, ordered=True)`, and that is not the same as having the name, because the pandas idiom is the one that appears in real code and in the board.
`Categorical` is a column type a caller can build without a series around it. `pd.Categorical(values, categories=..., ordered=...)` and `pd.Categorical.from_codes(codes, categories)` are both constructors, and the second one is how somebody who already has codes avoids a factorize. It also has `codes`, `categories`, `ordered`, `dtype` and most of what the accessor has, since the accessor is a thin thing over it in pandas.
`CategoricalIndex` is an index whose labels are a categorical. It is the one that unblocks a groupby reporting every category rather than only the ones with rows in them, since the result of `groupby(observed=False)` is indexed by one.
`CategoricalConversionWarning` is small and belongs with these, since it is what pandas raises reading a categorical out of Stata in chunks where the chunks disagree about what a code means. Firepanda has the same problem from Arrow and document 24 says what it does about it, so this is one name and a decision that already exists.
## Why they belong together
All four are the same missing idea, which is that a categorical is a type rather than a flag on a column. The accessor was worth doing first because it is what a caller reaches for at a prompt and because it is what the board measures most of, but every one of these four needs somewhere to put a list of categories and an ordering that is not a series, and building that once is the work.
The order to do them in is probably `CategoricalDtype` first, then `Categorical` over it, then `CategoricalIndex`, then the warning. `CategoricalDtype` alone unblocks `astype`, comparing two dtypes for equality, and `dtype ==` in user code, which is a lot of the value for the least of the work.
## What the board says
The board names `CategoricalDtype` on the module and 44 runs in `categorical/*`, some of which the accessor has now answered and some of which need this. A remeasure after [#432](https://github.com/tamnd/firepanda/pull/432) will say which is which, and that number should be in this issue before the work starts rather than guessed at now.
## Not in scope
Comparing two ordered categoricals with `<`, `value_counts` on a category column, and a groupby that reports the unused categories are all further out, and each is its own issue once the type exists.
Contributor guide
Research direction
Start with the categorical module and the 44 tests under categorical/*, then remeasure the board after #432 to identify what remains. Implement the requested pieces in the stated order—CategoricalDtype, Categorical, CategoricalIndex, and CategoricalConversionWarning—while keeping the listed comparison, value_counts, and unused-category groupby work out of scope.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data-engineering
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100