tamnd / tamnd/firepanda

D. Nested data: list, struct, explode and json_normalize

Open
#160 0 comments 0 reactions 0 assignees View on GitHub
area/dtype parity
Dominant language
Mojo
Stars
1
Forks
0
PR merge metrics
PR metrics pending

Description

pandas covers nested data thinly. The `.list` accessor is two methods and `.struct` is two plus a property, and that is the whole public surface, so the pandas denominator here is five callables and reaching 100 percent of it is a small job on top of the Arrow layout we already have.

The interesting part is what sits beside it. Polars has a much larger nested API, and anything we implement past the pandas five is an extension rather than a divergence. The report keeps those apart on purpose, because a library that adds a method has not failed to match anything, and a suite that counts extensions as conformance is inflating its own score.

### What it covers

`.list.flatten`, `.list.len`, `.struct.explode`, `.struct.field`, `.struct.dtypes`, plus `DataFrame.explode`, `Series.explode` and `json_normalize`.

### Done when

- [ ] The five pandas nested names at L3 100 percent
- [ ] `explode` on both types, including the empty list case which produces a null row rather than no row, and `ignore_index`
- [ ] `json_normalize` with `record_path`, `meta` and `sep`, since this is how nested JSON gets into a frame in practice
- [ ] Any extension beyond the pandas five is listed in the report as an extension, with the Polars name it matches

### How this is measured

The target above is an L3 rate over this workstream's callables, reported by `pixi run conformance` in [firepanda-compat](https://github.com/tamnd/firepanda-compat) and enforced by the CI ratchet rather than by a person ticking a box. L3 means every parameter takes every one of its values and the combinations that interact are enumerated. The levels are defined in [01-what-100-percent-means.md](https://github.com/tamnd/firepanda-compat/blob/main/docs/specs/01-what-100-percent-means.md), the counts come from [02-the-surface.md](https://github.com/tamnd/firepanda-compat/blob/main/docs/specs/02-the-surface.md), and the ordering argument for this milestone is in [08-m6.md](https://github.com/tamnd/firepanda-compat/blob/main/docs/specs/08-m6.md).

Anything we deliberately will not do goes in the divergence registry with a reason before this issue closes, and a registered divergence still runs and still has to diverge.

Part of #8, milestone M6.

Contributor guide

Open the contributing guide

Research direction

Start by running `pixi run conformance` and read the linked firepanda-compat specifications, especially 01-what-100-percent-means.md, 02-the-surface.md, and 08-m6.md. Use the issue's listed nested accessors, explode cases, and json_normalize parameters as the scope; done means the five pandas names reach L3 100 percent, required cases pass, and extensions or divergences are recorded.

Written by the indexing model from the issue text.

Assessment

Tech stack
pandas
Domain
data-engineering
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.