microsoft / microsoft/agent-framework

Python: .NET & Python: [Feature]: Externalize Durable Agents conversation storage to customer-owned stores

Open
#6,675 0 comments 0 reactions 0 assignees View on GitHub
.NET compaction durabletask python
Dominant language
Python
Stars
13.6k
Forks
2.3k
Avg merge
2d 45m
Merged PRs (30d)
358

Description

## Summary

The Durable Agents extension (`Microsoft.Agents.AI.DurableTask` for .NET, `agent-framework-durabletask` for Python) currently stores the full conversation history for every agent session inside Durable Task framework (DTF) entity state. This causes two adoption-blocking problems:

1. **Hard ~1 MB size ceiling.** The Durable Task Scheduler (DTS) limits a single entity payload to ~1 MB by default. Because each turn re-checkpoints the full conversation history, long-running sessions hit a wall. The `LargePayloadStorage` interceptor in `durabletask-dotnet` raises this to ~10 MB but does not remove the underlying unbounded-growth problem.
2. **No customer ownership of conversation data.** Conversations live only inside DTF entity state, opaque to the customer's data plane. Compliance, audit, and analytics customers need transcripts in storage they own and operate (e.g., their own Cosmos DB).

## Proposed solution

Introduce a pluggable `DurableAgentConversationStore` extension point that lets customers redirect durable-agent conversation history into a store they own, while preserving today's default in-entity-state behavior. Shipped implementations:

- `EntityStateConversationStore` — **default**, preserves today's behavior (exactly-once; the store *is* entity state).
- A **first-class backend store** (e.g., Cosmos) that persists an idempotency marker atomically with each turn — **recommended for production** (exactly-once).
- `ChatHistoryProviderConversationStore` / `HistoryProviderConversationStore` — a low-friction adapter bridging to any existing `ChatHistoryProvider` (.NET) / `HistoryProvider` (Python). Documented as **at-least-once** because the existing provider contract has no correlation-keyed backend primitive.

The contract centers on an atomic `CommitTurnAsync(correlationId, request, response)` primitive plus a `TryGetTurnAsync(correlationId)` replay check. Read-back is via a new `DurableTaskClient.GetAgentConversationHistoryAsync(sessionId)` extension method.

Full design, alternatives, and the idempotency analysis are captured in **ADR-0027** (PR to follow).

## Out of scope (tracked separately)

- Entity-level compaction (the contract reserves a `ReplaceAsync` primitive for it).
- Per-content externalization of large `AIContent` items.
- Per-tool checkpointing inside a turn (to close the tool-double-execution window).
- Reconciling .NET vs. Python entity-state schema-version handling.
- A reference-holding external `HistoryProvider` for Python (none exists today).

## Acceptance criteria

- [ ] ADR-0027 reviewed and accepted by deciders.
- [ ] .NET: `DurableAgentConversationStore` abstraction + default store + first-class backend store + bridge + `UseConversationStore` option + read-back extension + lazy migration + schema bump to `1.2.0`.
- [ ] Python parity for the storage abstraction and worker configuration.
- [ ] Unit + integration tests, including the at-least-once duplicate-window negative test and exactly-once recovery test.
- [ ] Documentation under `docs/features/durable-agents/`, including the `LargePayloadStorage` mitigation as a Phase-1 alternative.

## References

- `dotnet/src/Microsoft.Agents.AI.DurableTask/AgentEntity.cs` — current dual-storage entity.
- `dotnet/src/Microsoft.Agents.AI.Abstractions/ChatHistoryProvider.cs` — provider abstraction to bridge.
- `python/packages/durabletask/agent_framework_durabletask/_entities.py` — Python entity (same dual-storage pattern).

Contributor guide

Open the contributing guide

Research direction

Start by reading ADR-0027 and the current entity implementations in dotnet/src/Microsoft.Agents.AI.DurableTask/AgentEntity.cs and python/packages/durabletask/agent_framework_durabletask/_entities.py, then review the ChatHistoryProvider abstraction. Done means the proposed storage abstractions, backends, configuration, read-back, migration, tests, schema update, and durable-agent documentation meet the listed acceptance criteria.

Written by the indexing model from the issue text.

Assessment

Tech stack
csharp, python
Domain
backend, databases, distributed-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.