M8: Optimization pass
- Dominant language
- Mojo
- Stars
- 1
- Forks
- 0
- PR merge metrics
- PR metrics pending
Description
Tracking issue for **M8**. Specification: [`02-architecture.md`](../blob/main/docs/specs/02-architecture.md), [`03-dtype-dispatch.md`](../blob/main/docs/specs/03-dtype-dispatch.md) section 5.
The first milestone whose entire purpose is making existing things faster. It is scheduled here rather than earlier because by now there is a year of benchmark history saying where the time actually goes, instead of where we assumed it went.
### Scope
- [ ] The compile-time fused expression path, so that expressions written literally in Mojo source lower into a single loop with no intermediate buffers
- [ ] The native Parquet reader, replacing the M2 Arrow C++ binding: projection pushdown, predicate pushdown, row-group skipping, bloom filters
- [ ] Dictionary encoding end to end, so a group by on a string column never hashes a string
- [ ] Profile-driven kernel specialization
- [ ] **Whatever the compile-time and binary-size graph from M0 says needs doing.** If the curve has gone badly, narrowing the dtype lists happens here
### Exit criteria
- [ ] The fused path is at least 2x the interpreted path on a five-operation elementwise chain
- [ ] The native Parquet reader beats the Arrow C++ binding and matches Polars within 20%
- [ ] A group by on a dictionary-encoded string column is within 30% of the same group by on an integer column
- [ ] Compile time and binary size are inside their thresholds
### The Parquet reader has to justify itself
`firepanda-bench` measures Parquet twice after this milestone: once through the M2 binding and once through the native reader. If the two numbers come out the same, the native reader was not worth writing, and that is a finding rather than a failure.
### Depends on
M5.
Contributor guide
Research direction
Start with docs/specs/02-architecture.md and section 5 of docs/specs/03-dtype-dispatch.md, then review the M5 dependency and firepanda-bench measurements. The milestone is done when the fused path, native Parquet reader, dictionary encoding, specialization, and compile-time or binary-size thresholds meet the listed exit criteria.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp
- Domain
- data-engineering, performance
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100