tamnd / tamnd/firepanda

M2: Arrow interop and Parquet

Open
#4 11 comments 0 reactions 0 assignees View on GitHub
area/arrow area/io milestone
Dominant language
Mojo
Stars
1
Forks
0
Avg merge
1h 31m
Merged PRs (30d)
640

Description

Tracking issue for **M2**. Specification: [`08-milestones.md`](../blob/main/docs/specs/08-milestones.md), [`02-architecture.md`](../blob/main/docs/specs/02-architecture.md).

Deliberately early, because this is the milestone that makes every later gap survivable.

### Scope

- [x] Arrow C Data Interface declared: `ArrowSchema`, `ArrowArray`, the flags, the format string in both directions, the release protocol. A C function pointer usable as a struct field is `def (args) thin abi("C") -> None`, and a nullable C pointer is `Optional[Pointer[T, origin]]`, which is still one word because a null pointer is the niche
- [x] Export a column, no copying. `buffers` holds the column's own pointers, asserted by address identity. Strings are zero copy too, because firepanda's view layout is Arrow's byte for byte. Bool is packed on the way out, which is the one copy
- [x] Import a column. Copies, and the reasons are structural rather than temporary: firepanda buffers are 64-byte aligned and padded and Arrow's need not be, an Arrow array carries a row offset and a firepanda column starts at zero, and a foreign view array may have any number of data buffers where a firepanda column has one
- [x] Arrow IPC file and stream formats, both directions. `read_arrow` sniffs the magic number and reads either spelling; `write_arrow` and `write_ipc_stream` write them. String columns go out as views, because a firepanda string column is Arrow's view layout byte for byte and writing offsets would mean building an offset array and a compacted payload on every write. The body streams straight out of the column's buffers, so a 320 MB frame writes with kilobytes of allocation
- [x] Parquet by **binding DuckDB** through the C Data Interface, not by writing a reader. The spec said Arrow C++ and Arrow C++ exports no unmangled C symbols, so there is nothing for `dlsym` to find. DuckDB ships a stable C header with an Arrow export in it and brings globbing, Hive discovery and projection pushdown along with it. The dependency is soft: the library is `dlopen`ed on first use and a machine without it is a machine where `read_parquet` raises
- [x] Hive-partitioned dataset scanning through `ParquetOptions`. The partition columns come back as columns that are in no file, the projection pushes down onto them the same as onto a real one, and the directories a query cannot match are never opened. DuckDB detects a `key=value` tree by itself, so the setting is written into every query either way and the sayable thing is turning it off
- [x] NDJSON

### Exit criteria

- [ ] Every dtype round trips through pyarrow via the C Data Interface with zero copies, **verified by buffer pointer identity rather than by eye**
- [ ] TPC-H SF1 loads from Parquet
- [ ] A firepanda aggregation over TPC-H matches DuckDB exactly
- [ ] Parquet read through the binding is within 2x of Polars

That last bar is low on purpose. It is the right bar for a bound reader; the native one is M8.

### Why bind rather than implement

Binding keeps this milestone at roughly half the size it would otherwise be. A native Mojo Parquet reader is a *performance* project. A bound one is a *capability* that unblocks everything now.

### What this milestone actually buys

From here on, anything we have not implemented can be handed to DuckDB or pyarrow without copying. That converts incompleteness from a blocker into an inconvenience, and it is the highest-leverage risk reduction in the plan.

### Depends on

M1.

Contributor guide

Open the contributing guide

Research direction

Start with docs/specs/08-milestones.md and docs/specs/02-architecture.md, then inspect the existing read_arrow, write_arrow, write_ipc_stream, and ParquetOptions entry points mentioned in the issue. Run the pyarrow round-trip and TPC-H checks described in the exit criteria; done means every dtype passes pointer-identity checks, TPC-H matches DuckDB, and Parquet performance stays within 2x Polars.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data-engineering, databases
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.