tamnd / tamnd/firepanda

Write a list or a struct column to Arrow

Open
#284 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Mojo
Stars
1
Forks
0
PR merge metrics
PR metrics pending

Description

Follow-up to #282, which reads a list or a struct column and refuses to write one.

`_type_code` in `firepanda/io/arrow_ipc_write.mojo` raises `arrow ipc: cannot write a column of type list`, and that refusal is pinned by `test_a_nested_column_cannot_be_written_back_out_yet`. The refusal is the right behaviour for now, because the alternative is writing a list column's offsets buffer out as though it were a column of integers, and a reader would take that back as an int32 column with no complaint.

### What is missing

The writer has one type table per column and no children under it. A nested column needs four things it does not do:

- A type code and type table for `List`, `LargeList` and `Struct_`, which are tags 12, 21 and 13 in the Arrow type union. All three type tables are empty, so they are the easy part.
- Child fields in the schema table. A field in an Arrow schema has a `children` vector of fields, and a nested field's whole shape is in there. This is the part the writer has no shape for at all today.
- A recursive plan. `_plan` walks the columns and emits one field node and its buffers per column. A nested column emits its own node and then every node under it, in pre-order, matching the order the reader walks them in. A list node contributes an offsets buffer and a struct node contributes none.
- The buffer accounting that goes with it, since the reader takes the buffer vector positionally and a node that contributes the wrong number of buffers puts every column after it out of step.

The node list a column carries is already in pre-order, so the walk is the list in order and not a traversal. That is the reason the tree is held flat.

### Exit criteria

- [ ] A frame with a list, a large list, a struct and a struct of a list writes and reads back with the same values, the same offsets and the same offset widths
- [ ] The schema round trips byte identically, which is the third exit criterion of #270 extended to a nested column
- [ ] pyarrow reads what the writer produces, checked in the differential tests rather than only against this reader
- [ ] `test_a_nested_column_cannot_be_written_back_out_yet` is deleted rather than edited

### Depends on

#282.

Contributor guide

Open the contributing guide

Research direction

Read firepanda/io/arrow_ipc_write.mojo, especially _type_code and _plan, and review #282 before tracing the existing nested-column refusal in test_a_nested_column_cannot_be_written_back_out_yet. Use the named test and differential tests as the starting checks. Done means nested list and struct variants round-trip with matching values, offsets, widths, and byte-identical schemas, pyarrow accepts the output, and the refusal test is deleted.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data-engineering
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.