tamnd / tamnd/firepanda

H. Statistics and the rest of the frame

Open
#164 6 comments 0 reactions 0 assignees View on GitHub
area/frame area/kernel parity
Dominant language
Mojo
Stars
1
Forks
0
PR merge metrics
PR metrics pending

Description

61 callables that do not group into a namespace and are therefore easy to leave until last and then discover are half the reason people use pandas. `rank`, `cut`, `qcut`, `factorize`, `interpolate`, `replace`, `describe`, `corr`, `cov`, `duplicated`, `nunique`, `mode`, `quantile`, `clip`, `round`, `pct_change`, `diff`, `shift`, `where`, `mask`, `between`, `isin`, `nlargest`, `nsmallest`, `idxmin`, `idxmax`, `cummax` and the rest of that surface.

### The ones with real difficulty in them

`rank` has five methods and the `ties` behaviour is the whole feature. `cut` and `qcut` have bin edge inclusion rules that are off by one in the obvious implementation and produce a categorical, which ties this to the categoricals issue. `interpolate` has around a dozen methods and the linear one is not the hard part. `replace` accepts scalars, lists, dicts and nested dicts and the dict form is where the behaviour lives. `describe` returns a different set of rows per dtype and is the single most common thing a user types after loading a file, so being close is not enough.

### Done when

- [ ] `rank` with all five methods, `ascending` and `pct`, and `na_option`
- [ ] `cut` and `qcut` with the pandas bin edge rules, returning a proper categorical
- [ ] `describe` matching row for row per dtype, including the mixed dtype frame case
- [ ] `interpolate` with the methods that do not need scipy, and the ones that do registered as unimplemented rather than silently absent
- [x] `replace` in all four argument forms
- [ ] `corr` and `cov` with `method` and `min_periods`
- [ ] L3 at 90 percent

### How this is measured

The target above is an L3 rate over this workstream's callables, reported by `pixi run conformance` in [firepanda-compat](https://github.com/tamnd/firepanda-compat) and enforced by the CI ratchet rather than by a person ticking a box. L3 means every parameter takes every one of its values and the combinations that interact are enumerated. The levels are defined in [01-what-100-percent-means.md](https://github.com/tamnd/firepanda-compat/blob/main/docs/specs/01-what-100-percent-means.md), the counts come from [02-the-surface.md](https://github.com/tamnd/firepanda-compat/blob/main/docs/specs/02-the-surface.md), and the ordering argument for this milestone is in [08-m6.md](https://github.com/tamnd/firepanda-compat/blob/main/docs/specs/08-m6.md).

Anything we deliberately will not do goes in the divergence registry with a reason before this issue closes, and a registered divergence still runs and still has to diverge.

Part of #8, milestone M6.

Contributor guide

Open the contributing guide

Research direction

Start with the listed callables and run `pixi run conformance` in firepanda-compat to establish the current L3 coverage. Read the referenced specs `01-what-100-percent-means.md`, `02-the-surface.md`, and `08-m6.md`; done means the listed behaviors are covered, L3 reaches 90 percent, and any deliberate divergences are registered.

Written by the indexing model from the issue text.

Assessment

Tech stack
pandas
Domain
data
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.