tamnd / tamnd/firepanda

M3: The Python front door

Open
#5 10 comments 0 reactions 0 assignees View on GitHub
area/python milestone packaging release risk
Dominant language
Mojo
Stars
1
Forks
0
Avg merge
1h 31m
Merged PRs (30d)
640

Description

Tracking issue for **M3**. Specification: [`07-python-bindings.md`](../blob/main/docs/specs/07-python-bindings.md) in full, [`12-the-python-front-door-measured.md`](../blob/main/docs/specs/12-the-python-front-door-measured.md), which is document 07 checked against a running toolchain, and [`13-the-bound-type-is-not-a-dataframe.md`](../blob/main/docs/specs/13-the-bound-type-is-not-a-dataframe.md), which is the same exercise done to the type builder.

`pip install firepanda` has to work on a machine with no Mojo toolchain, no conda and no compiler. If it does not, the audience is people who already have Mojo installed, and that audience is too small for this project to be worth doing.

### What changed in this issue

This was nine scope bullets and one bold instruction saying to answer the licensing question before writing any binding code. It is now ten issues, and the change came out of measuring rather than out of scope creep. Document 12 built three probes against Mojo 1.0.0 and the results moved three things.

The distribution problem, which document 07 calls the hardest one in the specification, is technically solved and costs 2.9 MB. An extension links two Mojo runtime libraries which pull in two more, it does not link libpython at all, and the four of them vendored beside a 177 KB extension import and run on a stock Homebrew CPython 3.14 started with `env -i` and a `PATH` of `/usr/bin:/bin`. Those are the macOS arm64 figures and they are the ceiling of the three platforms rather than a representative sample: Linux needs one fewer library, because `libAsyncRTMojoBindings` does not appear there at all, and the self contained set is 2.02 MB on aarch64 and 2.12 MB on x86-64. So the licence question stops being a gate on the milestone and becomes a gate on publication, because none of the binding code changes depending on how it is answered. It still has to be asked, now, in parallel.

The Arrow crossing is much closer than document 07 assumes, because M2 already built the hard part. The C Data Interface is in the tree, the capsule wrapper around it is about forty lines, and the probe handed a firepanda column to `pyarrow.array` with the values buffer address matching on both sides. What is left is coverage, ownership and one struct.

Document 13 then moved a fourth thing, and it is the largest of the four. `PythonTypeBuilder` has four methods on it and none of them reaches a type slot or a property, the type it produces cannot be subclassed, and its instances have no `__dict__`, so `df["revenue"]`, `len(df)`, `df.shape` and `for row in df` cannot be written in Mojo in this toolchain by any route. The object a user holds is therefore a pure Python object holding a Mojo one, the Mojo bindings are a private calling convention underneath it, and the thin Python layer stops being a convenience that could be dropped and becomes the layer the pandas API lives in. That grows #199 by one output and one test and changes nothing about the ordering.

And the two items document 07 spends a paragraph each on are the two that do not work. A function registered with `def_function` can raise exactly one Python exception type and it is `Exception`, and setting a typed one by hand does not survive the binding wrapper. `PyErr_CheckSignals` is not exposed at all. Those are now the risky parts of M3 and they are scheduled early.

### Scope, in order

- [ ] #197 P0. May the Mojo runtime be redistributed inside a wheel. Runs in parallel with everything, gates only #206
- [x] #198 P1. The module skeleton and the self contained build, with the distribution claim turned from an afternoon into a CI job
- [x] #199 P2. The `BINDINGS` table, the generated registration, the generated Python wrapper class, the type stubs and the parity tests
- [x] #200 P3. One Mojo `Error` onto seven Python exception classes, in the Python layer because the Mojo layer cannot
- [ ] #201 P4. Ctrl-C, with no `PyErr_CheckSignals` to call
- [ ] #202 P5. The Arrow PyCapsule protocol, the array direction, every dtype and both ways. Frames done both ways, `PySeries` landed in #222, the series export is in #223
- [x] #203 P6. `ArrowArrayStream` and `__arrow_c_stream__`, the one Arrow piece firepanda does not have
- [ ] #204 P7. Release the GIL around execution, and the rule about worker threads
- [ ] #205 P8. The wheels, four platforms and a free threaded one, with the size gated
- [ ] #206 P9. Publish, to PyPI and to `modular-community`
- [x] #224 P10. Making a frame out of Python, because `pd.DataFrame({...})` is line one of everything. Added after the milestone was written, when it became clear the front door had no way in

The ordering argument is short. P1 goes first because every other issue needs somewhere to put its code, and because the distribution measurement is currently a fact about one afternoon on one machine rather than a property of the project. P2 goes second because the binding API is described by Modular as early and expected to change, and document 12 found two specific things that are likely to change in it, so the table has to exist before four hundred bindings do rather than after the first fifty have been hand written. P3 and P4 go next because they are the two with unknowns left in them and the milestone should find out early whether the recommended answers hold. P5 and P6 are the centrepiece by importance and not by risk, and P5 is nearly free. P7 is confirmed to work and is small. P8 is most of the remaining unknown, all of it on Linux and free threaded. P9 is last and is blocked on P0.

### Exit criteria

- [ ] The example in [`04-python-dx.md`](../blob/main/docs/specs/04-python-dx.md) section 2 runs from a clean virtualenv on macOS arm64 and manylinux x86-64, installed with `pip install firepanda`, with **no Mojo toolchain present**
- [x] `to_pandas`, `to_polars` and passing the frame to DuckDB all work with no copy, verified by buffer pointer identity on every dtype rather than on one
- [ ] Ctrl-C interrupts a running query and raises `KeyboardInterrupt`
- [x] The surface parity test passes over all three surfaces: every public Mojo method appears in the table and on the Python class, and vice versa in every direction
- [x] The signature parity test passes: the Python class exposes the pandas signature under `inspect.signature`, with defaults and keyword only arguments where pandas has them, rather than merely a method of the same name
- [x] The error mapping table is exercised by a test per row, asserting the class and not only the message
- [ ] The free threaded 3.14 wheel passes the same suite as the standard build **and** reports `sys._is_gil_enabled()` false, because a wheel that silently re-enables the GIL would pass the first half of that sentence while defeating the point
- [ ] Wheel size is recorded in CI with a threshold, starting near four megabytes

### Where each criterion stands today

Measured in document 12, before any of the issues above are started.

Both parity tests now pass and run on every platform in CI, over the five members that exist. They are ticked because the machinery is built and holds, not because the surface is finished, and they will keep being the thing that fails when it grows wrong. The error table is now built and tested a row at a time, and document 14 records both how the class gets across a boundary that cannot carry one and the three things about it that are still missing. The rest of this paragraph is as it was written before #199.

The clean virtualenv criterion is half met: the mechanism is proven end to end on macOS arm64 and on both Linux architectures in CI, and what is missing is the wheel and the example. Zero copy is met for the array protocol on one dtype, by address. Ctrl-C is blocked on #201. Both parity tests are not started and wait on #199. The free threaded wheel is untested and there is no free threaded interpreter on the machine that produced these numbers. Wheel size is not started.

### Frame this correctly internally

At M3 there is no lazy engine, no optimizer and no parallel execution. What ships is an eager columnar frame with fast CSV, fast Parquet and fast group by. That is enough to be interesting and **it is not enough to be impressive**, and the milestone has to be framed that way internally so that a lukewarm reception is read correctly.

### Depends on

M2.

Contributor guide

Open the contributing guide

Research direction

Start with docs/specs/07-python-bindings.md and the measured findings in docs/specs/12-the-python-front-door-measured.md and docs/specs/13-the-bound-type-is-not-a-dataframe.md, then choose one unchecked scope issue such as #201, #202, #204 or #205. Done is defined by the corresponding exit criterion, including clean pip installation, protocol coverage, Ctrl-C, free-threaded wheels or the recorded wheel-size threshold.

Written by the indexing model from the issue text.

Assessment

Tech stack
pandas, python
Domain
api, data
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.