tamnd / tamnd/firepanda

M1: Eager frame, kernels, hash table, CSV

Open
#3 44 comments 0 reactions 0 assignees View on GitHub
area/frame area/hash area/io area/kernel milestone
Dominant language
Mojo
Stars
1
Forks
0
Avg merge
1h 31m
Merged PRs (30d)
640

Description

Tracking issue for **M1**. Specification: [`08-milestones.md`](../blob/main/docs/specs/08-milestones.md), [`05-kernels.md`](../blob/main/docs/specs/05-kernels.md).

Correctness first. Kernels are vectorized from the first one, because in this language that is not a separate decision.

### Scope

- [x] `DataFrame` and `Series`; select, cast, filter, `head`, `tail`, `slice`, `take`
- [x] Multi-key sort with null placement
- [x] `firepanda/hash`: the open-addressing table with batch probe and radix partitioning
- [x] Group by with sum, mean, min, max, count, first, last
- [x] Group by with std, var, median, quantile, distinct count
- [x] Joins: inner, left, right, outer, semi, anti, cross
- [x] `concat` and the null-handling functions
- [x] CSV reader and writer with schema inference
- [x] The display layer
- [x] A scalar twin for **every** kernel, plus the differential fuzz harness
- [x] Stand up [`tamnd/firepanda-bench`](https://github.com/tamnd/firepanda-bench)
- [ ] Differential testing against pandas 3.0.5, in-process

### Exit criteria

- [ ] db-benchmark group-by and join queries produce results identical to DuckDB at 1M and 10M rows, with the harness forcing materialization the way the official suite does
- [ ] CSV reading beats `pandas.read_csv` with the pyarrow engine
- [ ] Basic aggregations beat pandas at 10M rows
- [x] Every kernel has a twin and the differential fuzz passes 100M cases
- [x] Group by beats a `Dict`-based implementation by enough to have justified writing our own

That last one is an assumption taken from the MojoFrame paper. It should be **measured, not trusted** — if our hash table is not meaningfully better than `Dict`, one of this project's premises is wrong and it is much cheaper to find that out here.

### Why two things start here rather than later

**The benchmark harness.** It can only measure CSV reading and a handful of aggregations at this point and that is fine. A harness created at the end of a project only ever proves what its author already believed.

**pandas differential testing.** Mojo imports pandas in the same process, so the conformance harness works from M1 with no bindings in place. In the Go sibling spec this required an entire bindings milestone to exist first. Use it.

### Depends on

M0.

Contributor guide

Open the contributing guide

Research direction

Start with docs/specs/08-milestones.md and docs/specs/05-kernels.md to understand the M1 scope. Then locate the in-process pandas differential harness and benchmark harness described in the issue; done means the listed differential, DuckDB, CSV, and aggregation exit criteria are satisfied.

Written by the indexing model from the issue text.

Assessment

Tech stack
pandas, python
Domain
data-engineering, performance, testing-qa
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.