microsoft / microsoft/agent-framework

Add Redis-backed checkpoint storage for distributed workflow persistence

Open
#2,401 0 comments 1 reaction 1 assignee View on GitHub

Nobody has claimed this yet.

.NET python question workflows
Dominant language
Python
Stars
13.6k
Forks
2.3k
Avg merge
2d 45m
Merged PRs (30d)
358

Description

## Problem
The Agent Framework currently supports only two checkpoint storage implementations:
- **In-memory** (ephemeral, not durable)
- **Flat file** (local-only, not scalable)

These options are useful for development and simple scenarios, but they limit adoption in production environments. Workflows often need to be resumed across processes, pods, or machines, which requires a distributed and durable checkpoint store.

## Proposal
Introduce a **RedisCheckpointStorage** implementation that persists workflow checkpoints in Redis. This will enable:
- **Distributed resilience**: Agents can resume workflows across multiple hosts or containers.
- **Durability with TTL**: Checkpoints can be retained for configurable lifetimes, preventing stale state.
- **Scalability**: Redis is widely used in cloud-native architectures and integrates well with Kubernetes, Azure Cache for Redis, and other managed services.
- **Cross-language parity**: Implementations will be provided in both **Python** and **C#**, ensuring developers can use Redis-backed persistence regardless of language choice.

## Implementation Plan
- **Python**
- Create `RedisCheckpointStorage` class, mirroring `InMemoryCheckpointStorage` and `FileCheckpointStorage`.
- Use `aioredis` for async operations.
- Support configurable options: Redis URL, key prefix, TTL.
- **C#**
- Create `RedisCheckpointStore` implementing `ICheckpointStore`.
- Integrate with `CheckpointManager` for workflow runs.
- Use `StackExchange.Redis` for connection pooling and resilience.
- **Shared Features**
- Serialize checkpoints as JSON for readability and debugging.
- Optional support for binary formats (MessagePack/Protobuf) for performance.
- Add unit and integration tests (docker-compose Redis).
- Provide documentation and sample workflows demonstrating checkpoint persistence and resumption.

## Impact
This enhancement will:
- Expand checkpoint storage options beyond local-only solutions.
- Improve production-readiness of the framework.
- Encourage community adoption by aligning with a widely used, battle-tested backend.
- Lay groundwork for additional backends (e.g., Azure Blob Storage, PostgreSQL, SQLite) via the same pluggable pattern.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.