tamnd / tamnd/firepanda

G. GroupBy completion, the 32 that did not land at M1

Open
#163 1 comment 0 reactions 0 assignees View on GitHub
area/frame area/hash parity
Dominant language
Mojo
Stars
1
Forks
0
Avg merge
1h 31m
Merged PRs (30d)
640

Description

The GroupBy namespace is 56 callables and M1 covered roughly 24 of them, the aggregations everyone uses. This issue is the other 32, which are the ones that separate a groupby that computes sums from a groupby a real program can rely on.

### What is left

The transform family, `cumsum` and its siblings within groups, `rank` within groups, `nth`, `head` and `tail`, `first` and `last` with their null skipping rules, `ngroup`, `cumcount`, `sample`, `filter`, `resample` which is M7, `ohlc`, `pct_change`, `diff`, `shift` with `freq` which is M7, `describe`, `quantile`, `sem`, `skew`, `corrwith`, `value_counts`, `size` against `count`, and the `pipe` and `apply` escape hatches.

`apply` deserves a note. It is the slow path by definition and it is also how users express anything the built in aggregations do not cover. It cannot be left out and it should not be optimised first.

### Done when

- [ ] The transform family returns a result aligned to the input rather than to the groups, which is the whole point of a transform and the easiest thing to get subtly wrong
- [ ] `as_index`, `sort`, `dropna` and `group_keys` behave on every method in the namespace rather than on the ones that were convenient
- [ ] `nth`, `head`, `tail`, `cumcount` and `ngroup`
- [ ] Multiple aggregations through `agg` with a dict, a list, and named aggregation, including the column naming rules
- [ ] `SpecificationError` where pandas raises it
- [ ] L3 at 90 percent

### How this is measured

The target above is an L3 rate over this workstream's callables, reported by `pixi run conformance` in [firepanda-compat](https://github.com/tamnd/firepanda-compat) and enforced by the CI ratchet rather than by a person ticking a box. L3 means every parameter takes every one of its values and the combinations that interact are enumerated. The levels are defined in [01-what-100-percent-means.md](https://github.com/tamnd/firepanda-compat/blob/main/docs/specs/01-what-100-percent-means.md), the counts come from [02-the-surface.md](https://github.com/tamnd/firepanda-compat/blob/main/docs/specs/02-the-surface.md), and the ordering argument for this milestone is in [08-m6.md](https://github.com/tamnd/firepanda-compat/blob/main/docs/specs/08-m6.md).

Anything we deliberately will not do goes in the divergence registry with a reason before this issue closes, and a registered divergence still runs and still has to diverge.

Part of #8, milestone M6.

Contributor guide

Open the contributing guide

Research direction

Start with `pixi run conformance` in firepanda-compat, then read `docs/specs/01-what-100-percent-means.md`, `02-the-surface.md`, and `08-m6.md` to understand the L3 target and callable scope. Use the GroupBy implementation entry points in the repository and the existing CI ratchet as guidance; done means 90% L3 coverage for this workstream, with deliberate divergences recorded in the divergence registry.

Written by the indexing model from the issue text.

Assessment

Tech stack
pandas, python
Domain
data, data-engineering
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
28/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.