microsoft / microsoft/agent-framework
Python: .NET & Python: [Feature]: Externalize Durable Agents conversation storage to customer-owned stores
- Dominant language
- Python
- Stars
- 13.6k
- Forks
- 2.3k
- Avg merge
- 2d 45m
- Merged PRs (30d)
- 358
Description
## Summary
The Durable Agents extension (`Microsoft.Agents.AI.DurableTask` for .NET, `agent-framework-durabletask` for Python) currently stores the full conversation history for every agent session inside Durable Task framework (DTF) entity state. This causes two adoption-blocking problems:
1. **Hard ~1 MB size ceiling.** The Durable Task Scheduler (DTS) limits a single entity payload to ~1 MB by default. Because each turn re-checkpoints the full conversation history, long-running sessions hit a wall. The `LargePayloadStorage` interceptor in `durabletask-dotnet` raises this to ~10 MB but does not remove the underlying unbounded-growth problem.
2. **No customer ownership of conversation data.** Conversations live only inside DTF entity state, opaque to the customer's data plane. Compliance, audit, and analytics customers need transcripts in storage they own and operate (e.g., their own Cosmos DB).
## Proposed solution
Introduce a pluggable `DurableAgentConversationStore` extension point that lets customers redirect durable-agent conversation history into a store they own, while preserving today's default in-entity-state behavior. Shipped implementations:
- `EntityStateConversationStore` — **default**, preserves today's behavior (exactly-once; the store *is* entity state).
- A **first-class backend store** (e.g., Cosmos) that persists an idempotency marker atomically with each turn — **recommended for production** (exactly-once).
- `ChatHistoryProviderConversationStore` / `HistoryProviderConversationStore` — a low-friction adapter bridging to any existing `ChatHistoryProvider` (.NET) / `HistoryProvider` (Python). Documented as **at-least-once** because the existing provider contract has no correlation-keyed backend primitive.
The contract centers on an atomic `CommitTurnAsync(correlationId, request, response)` primitive plus a `TryGetTurnAsync(correlationId)` replay check. Read-back is via a new `DurableTaskClient.GetAgentConversationHistoryAsync(sessionId)` extension method.
Full design, alternatives, and the idempotency analysis are captured in **ADR-0027** (PR to follow).
## Out of scope (tracked separately)
- Entity-level compaction (the contract reserves a `ReplaceAsync` primitive for it).
- Per-content externalization of large `AIContent` items.
- Per-tool checkpointing inside a turn (to close the tool-double-execution window).
- Reconciling .NET vs. Python entity-state schema-version handling.
- A reference-holding external `HistoryProvider` for Python (none exists today).
## Acceptance criteria
- [ ] ADR-0027 reviewed and accepted by deciders.
- [ ] .NET: `DurableAgentConversationStore` abstraction + default store + first-class backend store + bridge + `UseConversationStore` option + read-back extension + lazy migration + schema bump to `1.2.0`.
- [ ] Python parity for the storage abstraction and worker configuration.
- [ ] Unit + integration tests, including the at-least-once duplicate-window negative test and exactly-once recovery test.
- [ ] Documentation under `docs/features/durable-agents/`, including the `LargePayloadStorage` mitigation as a Phase-1 alternative.
## References
- `dotnet/src/Microsoft.Agents.AI.DurableTask/AgentEntity.cs` — current dual-storage entity.
- `dotnet/src/Microsoft.Agents.AI.Abstractions/ChatHistoryProvider.cs` — provider abstraction to bridge.
- `python/packages/durabletask/agent_framework_durabletask/_entities.py` — Python entity (same dual-storage pattern).
Contributor guide
Research direction
Start by reading ADR-0027 and the current entity implementations in dotnet/src/Microsoft.Agents.AI.DurableTask/AgentEntity.cs and python/packages/durabletask/agent_framework_durabletask/_entities.py, then review the ChatHistoryProvider abstraction. Done means the proposed storage abstractions, backends, configuration, read-back, migration, tests, schema update, and durable-agent documentation meet the listed acceptance criteria.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- csharp, python
- Domain
- backend, databases, distributed-systems
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100