tamnd / tamnd/firepanda

Join a nested column that arrives in more than one record batch

Open
#285 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Mojo
Stars
1
Forks
0
PR merge metrics
PR metrics pending

Description

Follow-up to #282, which reads a nested column that arrives in one record batch and refuses one spread across several by name.

The message is `arrow ipc: column 'x' is a nested type spread across 2 record batches, and firepanda reads a nested column that arrives in one`, and it is pinned by `test_a_nested_column_in_more_than_one_batch_is_refused_by_name` against a pyarrow fixture that writes exactly that.

### Why it is not just a concatenation

Every other column type joins across batches by laying the values end to end, which is what `assemble` does on every core. A list column cannot. Its offsets are positions into its own element column, so the second batch's offsets all have to be shifted by the number of elements the first batch's child holds, and the same is true at every level of a deep column. A struct joins by joining each field, which is the easy half, except that a struct of a list is both problems at once.

The refusal is by name because the alternative is handing back the first batch and calling it the file, or handing back offsets that point into the wrong elements, and neither of those looks wrong from the outside.

### What it needs

- A per node element count carried down the walk, since the shift for a node is the total length of the node below it in the batches already taken, not the row count.
- The join done node by node over the flat list, which the pre-order guarantees is a parent before its children.
- Validity joined the same way it is joined today for a plain column, at every node rather than only at the root.

The single batch path should stay the path a single batch takes, since one batch is the common case and it copies nothing.

### Exit criteria

- [ ] A list, a struct, a large list and a struct of a list each read from a three batch stream with the same values as from a one batch stream of the same frame
- [ ] The offsets of the second and third batches point at the elements those batches wrote, checked against the element column rather than against the offsets of another read
- [ ] `test_a_nested_column_in_more_than_one_batch_is_refused_by_name` is deleted rather than edited

### Depends on

#282.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.