tamnd / tamnd/firepanda

M5: Parallel execution

Open
#7 0 comments 0 reactions 0 assignees View on GitHub
area/exec milestone
Dominant language
Mojo
Stars
1
Forks
0
Avg merge
1h 31m
Merged PRs (30d)
640

Description

Tracking issue for **M5**. Specification: [`02-architecture.md`](../blob/main/docs/specs/02-architecture.md), [`09-quality-bar.md`](../blob/main/docs/specs/09-quality-bar.md) section 3.

### Scope

- [ ] Morsel-driven scheduler over `parallelize`, worker pool sized to physical core count, work stealing
- [ ] Partitioned hash aggregation: each worker owns a private table over a radix partition of the key space
- [ ] Parallel joins and sort
- [ ] Late materialization through selection vectors
- [ ] Cancellation flag threaded to morsel boundaries
- [ ] **Create `firepanda/exec/shared.mojo`** and put every atomic and mutable global in the library in it, each with its invariant
- [ ] Declare the operator-interface additions M9 and M10 need: device affinity and a streamable flag, both ignored for now
- [ ] Determine chunk size by benchmark in this milestone rather than guessing earlier

### Exit criteria

- [ ] Group by scales at least 12x on 16 cores at 100M rows, and better than 60% efficiency on 8 cores
- [ ] The concurrency stress suite passes ten thousand iterations under load
- [ ] Cancellation latency under 50ms at 100M rows
- [ ] `shared.mojo` accounts for every atomic and every mutable global in the library, checked by a grep in CI

### There is no race detector in this language

Go has `-race` and it would catch the entire class of bug this milestone produces. Mojo has nothing equivalent. The mitigation is structural rather than a test suite pretending to be a sanitizer:

- Workers own their partitions exclusively. Radix-partitioned aggregation is chosen partly because it needs **no shared mutable state at all**.
- Where a design has a lock-free and a shared variant with similar performance, take the one with no sharing.
- Every cancellation path gets an explicit test asserting every worker exits and no buffer is left owned. A morsel executor with a worker pool is exactly the shape that deadlocks or leaks on cancellation.

If a sanitizer becomes available in the toolchain, adopt it the week it lands.

### Depends on

M4.

Contributor guide

Open the contributing guide

Research direction

Start by reading docs/specs/02-architecture.md and section 3 of docs/specs/09-quality-bar.md, then inspect the existing parallelize path and the M4 implementation. The milestone spans scheduling, aggregation, joins, sorting, cancellation, and shared state, so the relevant entry points must be mapped before work begins. Done means the listed exit criteria pass, including the concurrency stress suite, cancellation latency, scaling targets, and CI's shared.mojo check.

Written by the indexing model from the issue text.

Assessment

Domain
data-engineering, distributed-systems, performance
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.