M5: Parallel execution
- Dominant language
- Mojo
- Stars
- 1
- Forks
- 0
- Avg merge
- 1h 31m
- Merged PRs (30d)
- 640
Description
Tracking issue for **M5**. Specification: [`02-architecture.md`](../blob/main/docs/specs/02-architecture.md), [`09-quality-bar.md`](../blob/main/docs/specs/09-quality-bar.md) section 3.
### Scope
- [ ] Morsel-driven scheduler over `parallelize`, worker pool sized to physical core count, work stealing
- [ ] Partitioned hash aggregation: each worker owns a private table over a radix partition of the key space
- [ ] Parallel joins and sort
- [ ] Late materialization through selection vectors
- [ ] Cancellation flag threaded to morsel boundaries
- [ ] **Create `firepanda/exec/shared.mojo`** and put every atomic and mutable global in the library in it, each with its invariant
- [ ] Declare the operator-interface additions M9 and M10 need: device affinity and a streamable flag, both ignored for now
- [ ] Determine chunk size by benchmark in this milestone rather than guessing earlier
### Exit criteria
- [ ] Group by scales at least 12x on 16 cores at 100M rows, and better than 60% efficiency on 8 cores
- [ ] The concurrency stress suite passes ten thousand iterations under load
- [ ] Cancellation latency under 50ms at 100M rows
- [ ] `shared.mojo` accounts for every atomic and every mutable global in the library, checked by a grep in CI
### There is no race detector in this language
Go has `-race` and it would catch the entire class of bug this milestone produces. Mojo has nothing equivalent. The mitigation is structural rather than a test suite pretending to be a sanitizer:
- Workers own their partitions exclusively. Radix-partitioned aggregation is chosen partly because it needs **no shared mutable state at all**.
- Where a design has a lock-free and a shared variant with similar performance, take the one with no sharing.
- Every cancellation path gets an explicit test asserting every worker exits and no buffer is left owned. A morsel executor with a worker pool is exactly the shape that deadlocks or leaks on cancellation.
If a sanitizer becomes available in the toolchain, adopt it the week it lands.
### Depends on
M4.
Contributor guide
Research direction
Start by reading docs/specs/02-architecture.md and section 3 of docs/specs/09-quality-bar.md, then inspect the existing parallelize path and the M4 implementation. The milestone spans scheduling, aggregation, joins, sorting, cancellation, and shared state, so the relevant entry points must be mapped before work begins. Done means the listed exit criteria pass, including the concurrency stress suite, cancellation latency, scaling targets, and CI's shared.mojo check.
Written by the indexing model from the issue text.
Assessment
- Domain
- data-engineering, distributed-systems, performance
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100