M1: Eager frame, kernels, hash table, CSV
- Dominant language
- Mojo
- Stars
- 1
- Forks
- 0
- Avg merge
- 1h 31m
- Merged PRs (30d)
- 640
Description
Tracking issue for **M1**. Specification: [`08-milestones.md`](../blob/main/docs/specs/08-milestones.md), [`05-kernels.md`](../blob/main/docs/specs/05-kernels.md).
Correctness first. Kernels are vectorized from the first one, because in this language that is not a separate decision.
### Scope
- [x] `DataFrame` and `Series`; select, cast, filter, `head`, `tail`, `slice`, `take`
- [x] Multi-key sort with null placement
- [x] `firepanda/hash`: the open-addressing table with batch probe and radix partitioning
- [x] Group by with sum, mean, min, max, count, first, last
- [x] Group by with std, var, median, quantile, distinct count
- [x] Joins: inner, left, right, outer, semi, anti, cross
- [x] `concat` and the null-handling functions
- [x] CSV reader and writer with schema inference
- [x] The display layer
- [x] A scalar twin for **every** kernel, plus the differential fuzz harness
- [x] Stand up [`tamnd/firepanda-bench`](https://github.com/tamnd/firepanda-bench)
- [ ] Differential testing against pandas 3.0.5, in-process
### Exit criteria
- [ ] db-benchmark group-by and join queries produce results identical to DuckDB at 1M and 10M rows, with the harness forcing materialization the way the official suite does
- [ ] CSV reading beats `pandas.read_csv` with the pyarrow engine
- [ ] Basic aggregations beat pandas at 10M rows
- [x] Every kernel has a twin and the differential fuzz passes 100M cases
- [x] Group by beats a `Dict`-based implementation by enough to have justified writing our own
That last one is an assumption taken from the MojoFrame paper. It should be **measured, not trusted** — if our hash table is not meaningfully better than `Dict`, one of this project's premises is wrong and it is much cheaper to find that out here.
### Why two things start here rather than later
**The benchmark harness.** It can only measure CSV reading and a handful of aggregations at this point and that is fine. A harness created at the end of a project only ever proves what its author already believed.
**pandas differential testing.** Mojo imports pandas in the same process, so the conformance harness works from M1 with no bindings in place. In the Go sibling spec this required an entire bindings milestone to exist first. Use it.
### Depends on
M0.
Contributor guide
Research direction
Start with docs/specs/08-milestones.md and docs/specs/05-kernels.md to understand the M1 scope. Then locate the in-process pandas differential harness and benchmark harness described in the issue; done means the listed differential, DuckDB, CSV, and aggregation exit criteria are satisfied.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- pandas, python
- Domain
- data-engineering, performance, testing-qa
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100