aws / aws/aws-durable-execution-sdk-java
[Feature]: Add a public API for custom extension operations
- Dominant language
- Java
- Stars
- 28
- Forks
- 11
- Avg merge
- 1d 8h
- Merged PRs (30d)
- 47
Description
### What would you like?
Split durable operations into two API categories:
1. **Primitive operations** are directly supported by the durable execution backend and owned by the SDK, such as step, chained invoke, wait, callback, and child context operations.
2. **Extension operations** are higher-level operations implemented by composing primitive operations, such as `waitForCondition`, `map`, `parallel`, `withRetry`, `waitForCallback`, and `dag`.
This feature request is only about making the second category extensible. Developers should be able to create reusable custom extension operations in a separate Maven module without modifying or rebuilding the SDK. It should not allow developers to add new backend operation types or implement primitive checkpoint protocols.
Extension libraries can expose operations as public static methods. Applications can statically import those methods and call them with ergonomics similar to native SDK operations.
A DAG operation is a concrete example. Under this extension model, it could be exposed from a separate DAG module as:
```java
import static software.amazon.lambda.durable.dag.DagOperations.dag;
DagResult result = dag("etl", d -> {
var a = d.step("a", String.class, (deps, step) -> "A");
var b = d.step("b", String.class, (deps, step) -> deps.get(a).orElseThrow() + "B")
.reads(a);
var c = d.step("c", String.class, (deps, step) -> deps.get(a).orElseThrow() + "C")
.reads(a);
d.step(
"join",
String.class,
(deps, step) -> deps.get(b).orElseThrow() + deps.get(c).orElseThrow())
.reads(b, c);
});
```
`DagOperations.dag` would retrieve the active context from SDK-managed thread-local storage through `DurableContext.getCurrentContext()`. This avoids adding `dag` and `dagAsync` to the core `DurableContext` interface and does not require extension registration or a typed context adapter.
Existing SDK operations can also be implemented or exposed as static methods backed by the same current-context mechanism. This allows primitive operations, built-in extension operations, and third-party extension operations to use one consistent style:
```java
import static software.amazon.lambda.durable.DurableOperations.map;
import static software.amazon.lambda.durable.DurableOperations.step;
var validated = step("validate", Input.class, stepContext -> validate(input));
var results = map("process", items, ItemResult.class, (item, index, context) -> process(item));
```
The existing instance methods on `DurableContext` can remain for compatibility. Static SDK methods can delegate to the active `DurableContext`, just as external extension methods do.
The SDK should provide:
- Stable public interfaces for composing extension operations from primitive operations.
- A supported extension scope with deterministic operation IDs for the primitives used internally.
- Reliable access to the active `DurableContext` from operation methods running on SDK-managed context threads.
- Synchronous and asynchronous extension forms consistent with built-in operations.
- A uniform static-import style that can be used by existing SDK operations and third-party extensions.
- Compatibility guarantees so extension implementations depend only on public SDK contracts.
The SDK would continue to own operation IDs, checkpoint/replay behavior, suspension, serialization, and communication with the backend for each primitive.
### Possible Implementation
The DAG extension could provide static entry points:
```java
public final class DagOperations {
private DagOperations() {}
public static DagResult dag(String name, Consumer register) {
return dagAsync(name, register, DagConfig.builder().build()).get();
}
public static DurableFuture dagAsync(
String name,
Consumer register,
DagConfig config) {
var context = DurableContext.getCurrentContext();
return DurableExtensions.runAsync(
context,
name,
DagResult.class,
extensionContext -> executeDag(extensionContext, register, config));
}
}
```
`DurableExtensions.runAsync(...)` is illustrative. The RFC should decide whether the SDK needs a dedicated extension runner or can stabilize an existing scoping primitive.
DAG demonstrates important requirements for that API. A DAG can run as a child-context container, delegate each task to existing durable operation machinery, and give tasks name-derived IDs so arbitrary graph scheduling remains replay-safe. The extension contract should support those requirements without exposing `DurableContextImpl`, adding operation-specific methods to `DurableContext`, or requiring an extension to send raw checkpoint updates.
An extension should receive an `ExtensionContext` exposing a stable set of primitive durable operations and extension-scope metadata. It should not expose `OperationUpdate`, backend polling, raw checkpoint state, internal execution managers, or the ability to define new backend operation types/subtypes.
Existing SDK operation methods could use the same structure internally: retrieve the current `DurableContext`, then delegate to the existing context-based implementation. This provides a common authoring and calling model without removing the current API.
The RFC should address:
- Which operations are primitives and therefore part of the stable extension contract.
- Which existing SDK operations should expose static facades and how those methods are organized and named.
- The contract and failure behavior of `DurableContext.getCurrentContext()` outside an active durable context or from a step thread.
- Thread-local context propagation across SDK-managed handler, child-context, executor, and virtual threads.
- Deterministic ID allocation and namespacing for primitives inside an extension.
- Stable named operation identities within an extension, as required by DAG tasks whose execution order may vary during replay.
- Sync and async execution and interoperability with `DurableFuture` combinators.
- Nested extensions and whether recursion or composition requires limits.
- Replay compatibility when an extension implementation changes between deployments.
- Serialization, exception, cancellation, completion, and plugin-hook behavior at the extension boundary.
- Whether existing SDK extension operations should be implemented on the same public contract.
Acceptance criteria:
- A separate Maven module can define and publish custom extension operations as static methods using only supported public APIs.
- Applications can statically import and invoke an extension operation without modifying `DurableContext`, registering a provider, or qualifying a utility class at every call site.
- Existing SDK primitive and extension operations can expose the same static-import calling style through the current-context mechanism.
- Existing `DurableContext` instance methods remain compatible.
- An operation method can reliably retrieve the current `DurableContext` while running on a supported SDK context thread.
- Calls outside a supported durable context fail with a clear, documented exception.
- Custom extensions can compose primitive operations but cannot define primitive/backend operation types or send raw checkpoint updates.
- Extension code does not subclass or reference `BaseDurableOperation`, `DurableContextImpl`, `ExecutionManager`, or other implementation details.
- Primitive operations inside an extension have deterministic, isolated IDs and replay correctly.
- Custom extensions work across initial execution, replay, and suspension/resume by relying on primitive operation semantics.
- Both synchronous and asynchronous extension usage are supported.
- `DurableFuture` combinators work with asynchronous extension results without internal downcasts.
- Nested extension behavior and plugin lifecycle behavior are documented and tested.
- Public extension interfaces have documented compatibility and versioning guarantees.
- A DAG extension can be implemented in a separate module and imported statically without adding `dag`/`dagAsync` to `DurableContext` or modifying `DurableContextImpl`.
- DAG task identities remain deterministic and replay-safe through supported public extension APIs rather than internal explicit-ID methods.
### Is this a breaking change?
No
### Does this require an RFC?
Yes
### Additional Context
This deliberately does not make the backend primitive-operation protocol extensible. The SDK remains responsible for backend-supported operation types and their checkpoint/replay implementation.
DAG is the motivating example. A core implementation would otherwise need DAG-specific entry points on `DurableContext`, implementation code in `DurableContextImpl`, and internal explicit-ID variants for primitive operations. This feature should stabilize the general capabilities needed by DAG so similarly complex extension operations can live outside the core SDK.
This is also related to #493, which tracks hard-coded operation types, and #492, which discussed optional features in extra packages.
Contributor guide
Assessment
This issue has not been assessed yet.