RFC: Agentic MapReduce / burst swarms for horizontal task parallelism

Summary

Add a Kimi-style burst execution primitive to jcode as a complement to its existing persistent swarm and DAG architecture.

The key distinction is between two kinds of multi-agent work:

• Persistent swarm agents are long-lived collaborators: they own tasks, maintain context, coordinate with peers, and participate in the evolving task DAG.
• Burst workers are ephemeral horizontal compute: many isolated subagents launched for a bounded map/scatter/review/race operation, then reduced back into a structured result for the owning task/agent.

The proposal is essentially agentic MapReduce embedded inside a jcode DAG node, not another competing swarm topology.

Motivation

jcode already has the hard coordination layer: persistent sessions, DAG/task ownership, worktrees, lifecycle/recovery, messaging, and repository-aware collaboration.

What seems missing is a lightweight way for one task to temporarily fan out into tens or hundreds of independent workers for embarrassingly parallel work — similar in spirit to Kimi's AgentSwarm horizontal scaling model.

The intended separation is:

DAG                 = what needs to happen
Persistent agents   = who owns the work
Burst workers       = how much parallel compute to throw at it

Proposed model

A persistent agent or DAG task could invoke something conceptually like:

swarm_burst(
    objective,
    items?,
    strategy = "map" | "scatter" | "review" | "race",
    workers = "auto",
    max_workers = 64,
    agent_profile?,
    model_policy?,
    tools?,
    write_policy = "none",
    reducer = "parent",
    budget?,
    timeout?
)

Example:

                 Coordinator
                      |
          +-----------+-----------+
          |           |           |
       Task A       Task B       Task C
    persistent    persistent    persistent
                      |
                 needs breadth
                      |
                burst_swarm()
                      |
       +--------------+--------------+
       |      |       |       |       |
      b1     b2      b3      ...     b64
       |      |       |               |
       +--------------+---------------+
                      |
                   reduce
                      |
                  Task B

Burst workers should not be ordinary swarm members

A burst worker should intentionally be lighter than a normal persistent jcode session:

persistent session?          no
appears in shared plan?      no
DM/channel participation?    no
can spawn descendants?       normally no
owns a long-lived worktree?  no
survives task completion?    no
returns structured result?   yes
can use tools?               yes
can read repo?               yes
can write main tree?         normally no

This avoids polluting jcode's durable swarm topology when a task briefly launches 50–300 speculative workers.

Make bursts children of tasks

Since jcode is moving toward a DAG-first model, burst execution fits naturally as an execution detail of a task node rather than as another durable agent hierarchy.

Possible runtime objects:

Burst
  burst_id
  task_id
  owner_session_id
  strategy
  worker_count
  worker_profile
  budget
  state

BurstWorker
  burst_id
  worker_id
  item
  state
  result_ref
  artifact_ref
  token_usage

The durable DAG remains readable even if a task launches 100 workers:

Task B
 └── Burst 17
      ├── Worker 1
      ├── Worker 2
      ├── ...
      └── Worker 100

Suggested primitives

Rather than treating every horizontal pattern as generic "swarm", a few explicit semantics could make the tool easier for agents to use correctly.

map

Run the same operation over independent items.

Examples:

• inspect every package/module
• analyze each failing test
• scan many files for a migration issue

scatter

Give the same problem to many independent workers and deliberately seek diverse hypotheses or approaches.

Examples:

• generate 32 independent root-cause hypotheses
• explore alternative architectures
• attempt multiple debugging strategies

review

Fan out independent critics over one candidate result.

Examples:

• security review
• correctness review
• performance review
• accessibility review
• API compatibility review

race

Run multiple solutions in parallel and stop once an acceptance predicate succeeds.

8 workers implement fix
        |
first candidate passing tests
        |
cancel remaining workers

Repository write isolation

Large burst swarms probably should not freely modify the shared repository.

Default behavior could be read-only:

worker
  -> read repo
  -> search / reason
  -> run sandboxed tools
  -> return findings

For coding attempts, workers could receive a temporary overlay, sandbox, or disposable worktree and return a patch/artifact. The owning persistent agent then decides what to adopt.

That preserves jcode's existing conflict/stale-read machinery for persistent collaborators without creating coordination noise from hundreds of speculative workers.

Structured results and reduction

A burst worker should return a compact structured result rather than dumping a full conversation into the parent's context, for example:

{
  "summary": "...",
  "confidence": 0.81,
  "findings": [],
  "files": [],
  "patch": null,
  "tests": [],
  "evidence": []
}

For large fan-out, support hierarchical reduction:

64 workers
   |
8 reducers
   |
2 reducers
   |
parent

The scalability challenge is not just launching 100+ workers; it is preventing 100+ raw outputs from becoming the next context bottleneck.

Persistent agents should be able to invoke bursts

This is where the two execution models become complementary.

Persistent Frontend Agent
        |
        ├── burst map x20
        |     "Find every dependency on the old table"
        |
        ├── performs implementation
        |
        └── burst review x8
              ├ accessibility
              ├ regressions
              ├ state handling
              ├ performance
              └ tests

Different persistent agents could each launch bursts independently while remaining responsible for integrating the results.

Scheduler / resource management

This likely belongs in the daemon rather than inside individual agents.

workers = auto could consider:

• provider request/token rate limits
• cost budget
• CPU / RAM
• sandbox slots
• worktree cost
• tool rate limits
• current persistent-agent load

Concurrency could ramp gradually instead of launching all workers at once.

Model routing

Burst execution is also a natural place for heterogeneous model routing:

burst:
  planner: expensive_reasoning_model
  workers:
    explore: cheap_fast_model
    coding: coding_model
    review: reviewer_model
  reducer: strong_synthesis_model

This allows expensive reasoning to remain concentrated in decomposition and integration while cheaper inference provides horizontal breadth.

UI

Burst workers should not explode the normal swarm graph.

A compact representation could look like:

● Frontend Agent
   └─ ⚡ 37/50 dependency scan
       ████████████░░░ 74%

● Backend Agent
   └─ ⚡ 6/8 implementation race

Individual workers could be inspectable on demand, while the main topology remains the persistent agents / task DAG.

Why MapReduce is a useful analogy

This is not classic data-processing MapReduce, but the analogy provides useful semantics:

Persistent agent
      |
    MAP / SCATTER
      |
many isolated tool-using agents
      |
 intermediate structured results
      |
     REDUCE
      |
persistent agent continues

The mapper is "agentic": it can reason, inspect the repository, run commands, call tools, and return artifacts. The reducer can itself be an agent.

A concise description of the feature is:

Agentic MapReduce embedded inside jcode's persistent swarm runtime.

Non-goals

• Replacing the existing persistent swarm model
• Turning hundreds of transient workers into first-class peers
• Giving all transient workers unconstrained shared-repository writes
• Requiring every DAG task to use burst execution

Open questions

1. Should burst workers be represented internally as lightweight sessions or a separate runtime type?
2. Should reduction be explicit (burst_reduce) or automatic as part of the burst invocation?
3. Is a temporary git worktree too expensive for large coding races; would overlays/snapshots be preferable?
4. Should burst workers be allowed to recursively burst, or should recursion initially be prohibited/budget-limited?
5. What should the default structured result schema be?
6. Should acceptance predicates for race be commands/tests, model judgments, or both?
7. How much of the persistent agent's context should a burst worker inherit versus receiving a purpose-built context snapshot?

Suggested rollout

A narrow first version could be:

1. burst_map only
2. read-only workers
3. fixed structured output
4. daemon-managed concurrency and cancellation
5. parent-agent reduction
6. compact UI status

Then add scatter, review, race, hierarchical reducers, model routing, and writable disposable sandboxes once the primitive is proven.

Begin by tracing the daemon's task/DAG execution and persistent swarm entry points described in the proposal. Scope the suggested first milestone: burst_map with read-only workers, fixed structured output, daemon-managed concurrency and cancellation, and parent-agent reduction; done means this narrow behavior is defined and integrated without changing the persistent swarm topology.