D. Nested data: list, struct, explode and json_normalize
- Dominant language
- Mojo
- Stars
- 1
- Forks
- 0
- PR merge metrics
- PR metrics pending
Description
pandas covers nested data thinly. The `.list` accessor is two methods and `.struct` is two plus a property, and that is the whole public surface, so the pandas denominator here is five callables and reaching 100 percent of it is a small job on top of the Arrow layout we already have.
The interesting part is what sits beside it. Polars has a much larger nested API, and anything we implement past the pandas five is an extension rather than a divergence. The report keeps those apart on purpose, because a library that adds a method has not failed to match anything, and a suite that counts extensions as conformance is inflating its own score.
### What it covers
`.list.flatten`, `.list.len`, `.struct.explode`, `.struct.field`, `.struct.dtypes`, plus `DataFrame.explode`, `Series.explode` and `json_normalize`.
### Done when
- [ ] The five pandas nested names at L3 100 percent
- [ ] `explode` on both types, including the empty list case which produces a null row rather than no row, and `ignore_index`
- [ ] `json_normalize` with `record_path`, `meta` and `sep`, since this is how nested JSON gets into a frame in practice
- [ ] Any extension beyond the pandas five is listed in the report as an extension, with the Polars name it matches
### How this is measured
The target above is an L3 rate over this workstream's callables, reported by `pixi run conformance` in [firepanda-compat](https://github.com/tamnd/firepanda-compat) and enforced by the CI ratchet rather than by a person ticking a box. L3 means every parameter takes every one of its values and the combinations that interact are enumerated. The levels are defined in [01-what-100-percent-means.md](https://github.com/tamnd/firepanda-compat/blob/main/docs/specs/01-what-100-percent-means.md), the counts come from [02-the-surface.md](https://github.com/tamnd/firepanda-compat/blob/main/docs/specs/02-the-surface.md), and the ordering argument for this milestone is in [08-m6.md](https://github.com/tamnd/firepanda-compat/blob/main/docs/specs/08-m6.md).
Anything we deliberately will not do goes in the divergence registry with a reason before this issue closes, and a registered divergence still runs and still has to diverge.
Part of #8, milestone M6.
Contributor guide
Research direction
Start by running `pixi run conformance` and read the linked firepanda-compat specifications, especially 01-what-100-percent-means.md, 02-the-surface.md, and 08-m6.md. Use the issue's listed nested accessors, explode cases, and json_normalize parameters as the scope; done means the five pandas names reach L3 100 percent, required cases pass, and extensions or divergences are recorded.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- pandas
- Domain
- data-engineering
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100