microsoft / microsoft/agent-framework
Add Redis-backed checkpoint storage for distributed workflow persistence
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 13.6k
- Forks
- 2.3k
- Avg merge
- 2d 45m
- Merged PRs (30d)
- 358
Description
## Problem
The Agent Framework currently supports only two checkpoint storage implementations:
- **In-memory** (ephemeral, not durable)
- **Flat file** (local-only, not scalable)
These options are useful for development and simple scenarios, but they limit adoption in production environments. Workflows often need to be resumed across processes, pods, or machines, which requires a distributed and durable checkpoint store.
## Proposal
Introduce a **RedisCheckpointStorage** implementation that persists workflow checkpoints in Redis. This will enable:
- **Distributed resilience**: Agents can resume workflows across multiple hosts or containers.
- **Durability with TTL**: Checkpoints can be retained for configurable lifetimes, preventing stale state.
- **Scalability**: Redis is widely used in cloud-native architectures and integrates well with Kubernetes, Azure Cache for Redis, and other managed services.
- **Cross-language parity**: Implementations will be provided in both **Python** and **C#**, ensuring developers can use Redis-backed persistence regardless of language choice.
## Implementation Plan
- **Python**
- Create `RedisCheckpointStorage` class, mirroring `InMemoryCheckpointStorage` and `FileCheckpointStorage`.
- Use `aioredis` for async operations.
- Support configurable options: Redis URL, key prefix, TTL.
- **C#**
- Create `RedisCheckpointStore` implementing `ICheckpointStore`.
- Integrate with `CheckpointManager` for workflow runs.
- Use `StackExchange.Redis` for connection pooling and resilience.
- **Shared Features**
- Serialize checkpoints as JSON for readability and debugging.
- Optional support for binary formats (MessagePack/Protobuf) for performance.
- Add unit and integration tests (docker-compose Redis).
- Provide documentation and sample workflows demonstrating checkpoint persistence and resumption.
## Impact
This enhancement will:
- Expand checkpoint storage options beyond local-only solutions.
- Improve production-readiness of the framework.
- Encourage community adoption by aligning with a widely used, battle-tested backend.
- Lay groundwork for additional backends (e.g., Azure Blob Storage, PostgreSQL, SQLite) via the same pluggable pattern.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.