aws / aws/aws-durable-execution-sdk-python

[Feature]: Add composable SerDes transformation pipelines

Open
#677 0 comments 0 reactions 0 assignees View on GitHub
enhancement needs-triage parity pkg:sdk
Dominant language
Python
Stars
53
Forks
25
Avg merge
1d 17h
Merged PRs (30d)
37

Description

## What would you like?

Add a composable SerDes processing pipeline while preserving the existing object-to-string `SerDes[T]` API as the value-codec boundary.

Today, transformations such as filesystem offloading, compression, encryption, signing, or custom envelopes must each wrap and own another complete `SerDes`. This makes independent transformations difficult to reuse or reorder.

The proposed model is:

```text
value -> root SerDes -> string stage -> string stage -> checkpoint
value <- root SerDes <- string stage <- string stage <- checkpoint
```

## Possible Implementation

- Introduce a synchronous `SerDesStage` protocol whose serialize/deserialize methods accept and return `str` plus `SerDesContext`.
- Add a `ComposableSerDes` or composition helper that starts with one existing `SerDes[T]`, applies stages in forward order during serialization, and reverses them during deserialization.
- Require stages to emit a self-identifying/versioned format. During deserialization, a stage should:
- reverse valid input in its own format;
- raise `SerDesError` for malformed input that identifies itself as that stage;
- return unrecognized input unchanged.
- Keep existing `SerDes` implementations and configuration APIs backward compatible.
- Allow `FileSystemSerDes` functionality to be exposed as a stage so callers can choose any root value codec and can place compression, encryption, or signing before or after storage as appropriate.
- Add tests for stage order, pass-through of external invocation input, null/empty values, malformed recognized envelopes, and custom value types.

## Is this a breaking change?

No.

## Does this require an RFC?

Yes.

## Additional Context

Python's current `FileSystemSerDes` already supports a configurable inner value SerDes. A pipeline would separate that value-codec responsibility from reusable string transformations instead of adding another nested-wrapper layer. A comparable Java design is being developed in aws/aws-durable-execution-sdk-java#648.

Contributor guide

Open the contributing guide

Research direction

Start by reading the existing SerDes and FileSystemSerDes implementations and their configuration APIs to understand the current value-codec boundary. Define the stage protocol and composition semantics through the required RFC, then add tests covering ordering, pass-through input, null and empty values, malformed recognized envelopes, and custom value types while preserving backward compatibility.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.