Series.corr and Series.cov need a second column and index alignment
- Dominant language
- Mojo
- Stars
- 1
- Forks
- 0
- PR merge metrics
- PR metrics pending
Description
Two of the seventeen reductions in `AggKind` were left out of #330 for the same reason, and it is not that the kernel is missing. `AggKind.CORR` and `AggKind.COV` are both implemented. What is missing is the shape of the call.
Every other reduction in that PR takes one column and answers one number, which is why they all fit through one boundary method that takes a reduction name and a float parameter. `corr` and `cov` take a second series:
```python
s.corr(other, method="pearson", min_periods=None)
s.cov(other, min_periods=None, ddof=1)
```
That second argument is not a number the existing door can carry. It is a whole series, and pandas aligns it against the receiver's index before computing anything, so `pd.Series([1, 2, 3], index=["a", "b", "c"]).corr(pd.Series([3, 2, 1], index=["c", "b", "a"]))` is not the same as pairing the two by position. Rows that appear in one index and not the other are dropped from the pair rather than treated as missing.
So this needs, in order:
- a boundary method that accepts a second `PySeries` rather than a scalar
- index alignment between two series, which is the piece that does not exist yet and is the actual work
- the pairwise missing rule, where a row is used only if both sides have a value there
`min_periods` and `method` are separate. `method` accepts `"pearson"`, `"kendall"`, `"spearman"` or a callable in pandas, and the kernel implements Pearson, so the other three should refuse by name the way the other unimplemented parameters in #330 do.
`DataFrame.corr` and `DataFrame.cov` answer a square frame rather than a number and are a further step again, but they sit on the same alignment work.
The conformance board already has `stats/corr-pearson`, `stats/cov` and `stats/corr-with-nulls` passing through the hand written driver, so the behaviour is measured. What is not measured is the same operation reached by a pandas program, which is what `resolution/series.corr` and `signature/series.corr` ask and currently report as absent.
Follow-up to #330. Part of the M6 work tracked in #8.
Contributor guide
Research direction
Start with the existing Series reduction boundary from #330, then inspect the conformance entries for resolution/series.corr and signature/series.corr and the hand-written driver results. Trace how a second PySeries could be passed and where index alignment and pairwise missing-value handling belong. Done means pandas-reached corr and cov match the measured Pearson behavior, while unsupported methods are refused by name.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- pandas, python
- Domain
- data-engineering
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 55/100