tamnd / tamnd/firepanda

C. Categoricals

Open
#159 1 comment 0 reactions 0 assignees View on GitHub
area/dtype parity
Dominant language
Mojo
Stars
1
Forks
0
PR merge metrics
PR metrics pending

Description

Eight `.cat` methods, plus `CategoricalDtype`, plus the `observed` and `ordered` behaviour that leaks into groupby, sorting and comparison. Small in callables and larger than it looks in consequences, because a categorical changes the answer of operations that do not mention categories anywhere in their signature.

### What it covers

The eight: `add_categories`, `as_ordered`, `as_unordered`, `remove_categories`, `remove_unused_categories`, `rename_categories`, `reorder_categories`, `set_categories`.

`observed=False` on a groupby over a categorical key produces a row for every category that never appeared, which is the behaviour most likely to be missed and most likely to change a result silently when it is. Sorting an ordered categorical sorts by category order rather than by value. Comparing two categoricals with different categories raises. Concatenating two categoricals with different categories produces the union, or an object column in older pandas, and pandas 3.0 is the behaviour we match.

### Done when

- [ ] The eight methods, with the null handling that `remove_categories` implies since removing a category turns its values into nulls
- [ ] `CategoricalDtype` with `categories` and `ordered`, and dtype equality that accounts for both
- [ ] `observed` on groupby, both values, which is the interaction to test hardest
- [ ] Ordered comparison and sorting by category order
- [ ] Concat and merge across differing category sets
- [ ] L3 at 100 percent

### How this is measured

The target above is an L3 rate over this workstream's callables, reported by `pixi run conformance` in [firepanda-compat](https://github.com/tamnd/firepanda-compat) and enforced by the CI ratchet rather than by a person ticking a box. L3 means every parameter takes every one of its values and the combinations that interact are enumerated. The levels are defined in [01-what-100-percent-means.md](https://github.com/tamnd/firepanda-compat/blob/main/docs/specs/01-what-100-percent-means.md), the counts come from [02-the-surface.md](https://github.com/tamnd/firepanda-compat/blob/main/docs/specs/02-the-surface.md), and the ordering argument for this milestone is in [08-m6.md](https://github.com/tamnd/firepanda-compat/blob/main/docs/specs/08-m6.md).

Anything we deliberately will not do goes in the divergence registry with a reason before this issue closes, and a registered divergence still runs and still has to diverge.

Part of #8, milestone M6.

Contributor guide

Open the contributing guide

Research direction

Start by reading the linked firepanda-compat specifications, especially 01-what-100-percent-means.md, 02-the-surface.md, and 08-m6.md, then run `pixi run conformance`. Use the eight listed categorical methods and the stated groupby, sorting, comparison, concat, merge, and dtype behaviors as the scope. Done means L3 reaches 100 percent, with deliberate divergences registered and still tested.

Written by the indexing model from the issue text.

Assessment

Tech stack
pandas
Domain
data
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.