aws / aws/aws-durable-execution-sdk-python

[Feature]: Support incremental composition of parallel operations

Aperta
#614 1 commento 0 reazioni 0 assegnatari Vedi su GitHub
enhancement parity
Lingua principale
Python
Stelle
53
Fork
25
Merge medio
1g 17h
PR unite (30g)
37

Descrizione

## What would you like?

Please add an incremental composition API for Durable Execution `parallel`
operations.

The current Python API requires the complete sequence of branch functions when
`context.parallel(...)` is called:

```python
result = context.parallel(
[run_research, run_analysis],
name="agent-tasks",
)
```

This is convenient when every branch is known up front, but it does not let a
workflow create a parallel operation, register and start branches as work is
discovered, and then explicitly seal and await the operation.

Incremental composition is especially useful for agentic AI workloads. An
agent may checkpoint a generated plan, discover tool calls or specialist tasks
from that plan, and add branches as those tasks become known. Ready work could
start immediately, subject to `max_concurrency`, while later branches are
still being assembled.

The Java Durable Execution SDK already exposes this lifecycle:

```java
ParallelDurableFuture parallel = context.parallel("agent-tasks");
try (parallel) {
DurableFuture research =
parallel.branch("research", ResearchResult.class, branch -> runResearch(branch));

if (plan.requiresReview()) {
DurableFuture review =
parallel.branch("review", ReviewResult.class, branch -> runReview(branch));
}
}
```

`parallel.branch(...)` registers and starts each branch, and closing/getting the
parent seals registration and waits for completion.

Benefits include:

1. Branches can be registered and started as a workflow plan is constructed.
2. Ready work need not wait for the entire branch sequence to be materialized.
3. Conditional and dynamic fan-out can use normal control flow without first
building an intermediate list.
4. Individual branch handles can compose naturally with later workflow logic.
5. The programming model supports agentic orchestration and other workloads
where the amount of work emerges incrementally.
6. It improves cross-SDK parity with Java.

The existing sequence-based `context.parallel(...)` API should remain available
for the common case where all branches are predefined.

## Possible Implementation

An additive builder/handle API could provide a registration phase followed by
an explicit completion phase. The exact naming is open for discussion; this is
only an illustrative Python shape:

```python
parallel = context.create_parallel(
name="agent-tasks",
config=ParallelConfig(max_concurrency=4),
)

research = parallel.branch("research", run_research)

if plan.requires_review:
review = parallel.branch("review", run_review)

summary = parallel.complete()
research_result = research.get()
```

A context-manager variant could also make the completion boundary explicit:

```python
with context.parallel_builder("agent-tasks") as parallel:
research = parallel.branch("research", run_research)
if plan.requires_review:
review = parallel.branch("review", run_review)

research_result = research.get()
```

The API should ideally:

* allow branches to be registered until `complete()` or context-manager exit
seals the operation
* allow registered branches to start immediately, respecting
`max_concurrency`
* preserve branch names and registration order for deterministic replay
* retain existing completion strategies, serialization, summary generation,
and per-branch error behavior
* reject branch registration after completion has begun
* preserve the existing sequence-based API as a convenience wrapper

Incremental registration must not weaken Durable Execution's deterministic
replay contract. For agentic workloads, model-generated plans or tool-call
sets should first be produced in a checkpointed step. Replay should then
register the same branches in the same order, with the SDK detecting
inconsistent registration.

## Is this a breaking change?

No. This can be introduced as an additive API.

## Does this require an RFC?

Yes. The registration lifecycle, completion semantics, replay validation, and
interaction with early completion strategies should be specified.

## Additional Context

References:

* [Current Python `DurableContext.parallel` API](https://github.com/aws/aws-durable-execution-sdk-python/blob/main/packages/aws-durable-execution-sdk-python/src/aws_durable_execution_sdk_python/types.py)
* [Java `ParallelDurableFuture`](https://github.com/aws/aws-durable-execution-sdk-java/blob/main/sdk/src/main/java/software/amazon/lambda/durable/ParallelDurableFuture.java)
* [Related .NET feature request](https://github.com/aws/aws-lambda-dotnet/issues/2519)

Guida per i contributori

Apri la guida per i contributori

Direzione di ricerca

Inizia con l'attuale API Python DurableContext.parallel in packages/aws-durable-execution-sdk-python/src/aws_durable_execution_sdk_python/types.py, quindi confronta il ciclo di vita esposto da ParallelDurableFuture di Java. Specifica in un RFC la semantica della registrazione, del completamento, della convalida del replay e del completamento anticipato prima dell'implementazione. Il lavoro è completato quando esiste un design API additivo concordato che preserva l'API esistente basata sulle sequenze e il contratto di replay deterministico.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Valutazione

Stack tecnologico
java, python
Ambito
backend-api-design, distributed-systems
Tipo di issue
Funzionalità
Difficoltà
5/5
Tempo stimato
Più di una settimana
Stato di attività
Tranquilla
Chiarezza
Da chiarire
Idoneità per principianti
35/100

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.