tamnd / tamnd/firepanda

M8: Optimization pass

Open
#10 0 comments 0 reactions 0 assignees View on GitHub
area/io area/kernel milestone performance
Dominant language
Mojo
Stars
1
Forks
0
PR merge metrics
PR metrics pending

Description

Tracking issue for **M8**. Specification: [`02-architecture.md`](../blob/main/docs/specs/02-architecture.md), [`03-dtype-dispatch.md`](../blob/main/docs/specs/03-dtype-dispatch.md) section 5.

The first milestone whose entire purpose is making existing things faster. It is scheduled here rather than earlier because by now there is a year of benchmark history saying where the time actually goes, instead of where we assumed it went.

### Scope

- [ ] The compile-time fused expression path, so that expressions written literally in Mojo source lower into a single loop with no intermediate buffers
- [ ] The native Parquet reader, replacing the M2 Arrow C++ binding: projection pushdown, predicate pushdown, row-group skipping, bloom filters
- [ ] Dictionary encoding end to end, so a group by on a string column never hashes a string
- [ ] Profile-driven kernel specialization
- [ ] **Whatever the compile-time and binary-size graph from M0 says needs doing.** If the curve has gone badly, narrowing the dtype lists happens here

### Exit criteria

- [ ] The fused path is at least 2x the interpreted path on a five-operation elementwise chain
- [ ] The native Parquet reader beats the Arrow C++ binding and matches Polars within 20%
- [ ] A group by on a dictionary-encoded string column is within 30% of the same group by on an integer column
- [ ] Compile time and binary size are inside their thresholds

### The Parquet reader has to justify itself

`firepanda-bench` measures Parquet twice after this milestone: once through the M2 binding and once through the native reader. If the two numbers come out the same, the native reader was not worth writing, and that is a finding rather than a failure.

### Depends on

M5.

Contributor guide

Open the contributing guide

Research direction

Start with docs/specs/02-architecture.md and section 5 of docs/specs/03-dtype-dispatch.md, then review the M5 dependency and firepanda-bench measurements. The milestone is done when the fused path, native Parquet reader, dictionary encoding, specialization, and compile-time or binary-size thresholds meet the listed exit criteria.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
data-engineering, performance
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.