Write a list or a struct column to Arrow
- Dominant language
- Mojo
- Stars
- 1
- Forks
- 0
- PR merge metrics
- PR metrics pending
Description
Follow-up to #282, which reads a list or a struct column and refuses to write one.
`_type_code` in `firepanda/io/arrow_ipc_write.mojo` raises `arrow ipc: cannot write a column of type list`, and that refusal is pinned by `test_a_nested_column_cannot_be_written_back_out_yet`. The refusal is the right behaviour for now, because the alternative is writing a list column's offsets buffer out as though it were a column of integers, and a reader would take that back as an int32 column with no complaint.
### What is missing
The writer has one type table per column and no children under it. A nested column needs four things it does not do:
- A type code and type table for `List`, `LargeList` and `Struct_`, which are tags 12, 21 and 13 in the Arrow type union. All three type tables are empty, so they are the easy part.
- Child fields in the schema table. A field in an Arrow schema has a `children` vector of fields, and a nested field's whole shape is in there. This is the part the writer has no shape for at all today.
- A recursive plan. `_plan` walks the columns and emits one field node and its buffers per column. A nested column emits its own node and then every node under it, in pre-order, matching the order the reader walks them in. A list node contributes an offsets buffer and a struct node contributes none.
- The buffer accounting that goes with it, since the reader takes the buffer vector positionally and a node that contributes the wrong number of buffers puts every column after it out of step.
The node list a column carries is already in pre-order, so the walk is the list in order and not a traversal. That is the reason the tree is held flat.
### Exit criteria
- [ ] A frame with a list, a large list, a struct and a struct of a list writes and reads back with the same values, the same offsets and the same offset widths
- [ ] The schema round trips byte identically, which is the third exit criterion of #270 extended to a nested column
- [ ] pyarrow reads what the writer produces, checked in the differential tests rather than only against this reader
- [ ] `test_a_nested_column_cannot_be_written_back_out_yet` is deleted rather than edited
### Depends on
#282.
Contributor guide
Research direction
Read firepanda/io/arrow_ipc_write.mojo, especially _type_code and _plan, and review #282 before tracing the existing nested-column refusal in test_a_nested_column_cannot_be_written_back_out_yet. Use the named test and differential tests as the starting checks. Done means nested list and struct variants round-trip with matching values, offsets, widths, and byte-identical schemas, pyarrow accepts the output, and the refusal test is deleted.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data-engineering
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100