A dictionary column cannot cross the Arrow C Data Interface
- Dominant language
- Mojo
- Stars
- 1
- Forks
- 0
- PR merge metrics
- PR metrics pending
Description
A dictionary encoded column can now be read from an Arrow IPC file or stream and written back to one, but it cannot cross the C Data Interface in either direction. This issue is that gap.
### Where it stands
`arrow_import.mojo` refuses an incoming array whose `dictionary` pointer is set, by name and with the reason in the message. `format_for` in `arrow_c.mojo` has no entry for a dictionary type, so exporting a categorical frame raises `arrow: no format string for category`, which is true but says nothing a caller can act on. `ArrowSchema` and `ArrowArray` already carry the `dictionary` field and `ARROW_FLAG_DICTIONARY_ORDERED` is already declared, so the structs need nothing.
### Why it was left out rather than done with the rest
The two transports disagree about where the categories live, and that disagreement is the whole of the work.
In IPC the categories arrive in dictionary batch messages of their own, keyed by id, once for the whole stream. That is the shape a firepanda column has, one set of categories per column, so the reader attaches them and nothing has to be reconciled.
On the C Data Interface every `ArrowArray` carries its own `dictionary` child. One column delivered as three batches is three arrays each with its own categories, and they are allowed to differ. A consumer that wants one column out of that has to either prove the three agree or unify them and rewrite the codes, and neither is a thing the import path does today.
The export half has the mirror image of the problem and an easier version of it, since firepanda holds one set of categories and can hand the same values array to every batch it produces.
### Scope
- [ ] `format_for` returns the index format for a dictionary type, since the C interface puts the index type in the format string and the values type in the `dictionary` child schema, which is the same split the IPC schema message makes
- [ ] `export_schema` fills the `dictionary` field with a child schema for the categories and sets `ARROW_FLAG_DICTIONARY_ORDERED` when the type is ordered
- [ ] `export_array` fills the `dictionary` field with the values array, with the release callback owning it the way the struct parent already owns its children
- [ ] Zero copy for the codes, and a decision recorded either way for the categories, since a values array shared by several exported batches cannot be released by the first of them to go
- [ ] `arrow_import.mojo` accepts a single array with a dictionary child and builds a categorical column from it
- [ ] A stream of several batches is either accepted when every batch agrees on the categories or refused by name when they do not, and the choice between that and unifying them is made here rather than assumed
- [ ] An out of range code is refused on this path too, with the same message shape the IPC path uses, which names the row and the code
- [ ] A round trip through pyarrow in both directions, asserting the reconstructed `CategoricalDtype` keeps its category order, its ordered flag and any unused category
### Not in scope
Dictionaries over anything but strings. firepanda's categories are strings, which is what pandas' are, and the IPC reader refuses the rest by name. Whatever this issue does should refuse them the same way rather than opening a second question.
Follows the dictionary column type. Part of #202.
Contributor guide
Research direction
Start with arrow_import.mojo and arrow_c.mojo, tracing format_for, export_schema, export_array, and the existing IPC dictionary path. Define the handling for shared category arrays and differing batch dictionaries before implementing the C Data Interface fields. Done means pyarrow round trips preserve category order, the ordered flag, unused categories, and reject unsupported or invalid inputs by name.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- pandas, python
- Domain
- data-engineering
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 52/100