G. GroupBy completion, the 32 that did not land at M1
- Dominant language
- Mojo
- Stars
- 1
- Forks
- 0
- Avg merge
- 1h 31m
- Merged PRs (30d)
- 640
Description
The GroupBy namespace is 56 callables and M1 covered roughly 24 of them, the aggregations everyone uses. This issue is the other 32, which are the ones that separate a groupby that computes sums from a groupby a real program can rely on.
### What is left
The transform family, `cumsum` and its siblings within groups, `rank` within groups, `nth`, `head` and `tail`, `first` and `last` with their null skipping rules, `ngroup`, `cumcount`, `sample`, `filter`, `resample` which is M7, `ohlc`, `pct_change`, `diff`, `shift` with `freq` which is M7, `describe`, `quantile`, `sem`, `skew`, `corrwith`, `value_counts`, `size` against `count`, and the `pipe` and `apply` escape hatches.
`apply` deserves a note. It is the slow path by definition and it is also how users express anything the built in aggregations do not cover. It cannot be left out and it should not be optimised first.
### Done when
- [ ] The transform family returns a result aligned to the input rather than to the groups, which is the whole point of a transform and the easiest thing to get subtly wrong
- [ ] `as_index`, `sort`, `dropna` and `group_keys` behave on every method in the namespace rather than on the ones that were convenient
- [ ] `nth`, `head`, `tail`, `cumcount` and `ngroup`
- [ ] Multiple aggregations through `agg` with a dict, a list, and named aggregation, including the column naming rules
- [ ] `SpecificationError` where pandas raises it
- [ ] L3 at 90 percent
### How this is measured
The target above is an L3 rate over this workstream's callables, reported by `pixi run conformance` in [firepanda-compat](https://github.com/tamnd/firepanda-compat) and enforced by the CI ratchet rather than by a person ticking a box. L3 means every parameter takes every one of its values and the combinations that interact are enumerated. The levels are defined in [01-what-100-percent-means.md](https://github.com/tamnd/firepanda-compat/blob/main/docs/specs/01-what-100-percent-means.md), the counts come from [02-the-surface.md](https://github.com/tamnd/firepanda-compat/blob/main/docs/specs/02-the-surface.md), and the ordering argument for this milestone is in [08-m6.md](https://github.com/tamnd/firepanda-compat/blob/main/docs/specs/08-m6.md).
Anything we deliberately will not do goes in the divergence registry with a reason before this issue closes, and a registered divergence still runs and still has to diverge.
Part of #8, milestone M6.
Contributor guide
Research direction
Start with `pixi run conformance` in firepanda-compat, then read `docs/specs/01-what-100-percent-means.md`, `02-the-surface.md`, and `08-m6.md` to understand the L3 target and callable scope. Use the GroupBy implementation entry points in the repository and the existing CI ratchet as guidance; done means 90% L3 coverage for this workstream, with deliberate divergences recorded in the divergence registry.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- pandas, python
- Domain
- data, data-engineering
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 28/100