microsoft / microsoft/agent-framework

.NET: [Bug]: DurableTask: SuperstepState.AccumulatedEvents overflows CustomStatus 16 KB cap on multi-executor workflows with typed outputs

Open
#5,745 4 comments 1 reaction 1 assignee Claimed by @kshyju View on GitHub
.NET
Dominant language
Python
Stars
13.6k
Forks
2.3k
Avg merge
2d 45m
Merged PRs (30d)
358

Description

## Summary

For Durable Functions workflows built on `Microsoft.Agents.AI.DurableTask`, the orchestrator-level `SuperstepState.AccumulatedEvents` accumulates serialized executor events across every superstep and projects the cumulative list into Durable Functions' `CustomStatus` via `SetCustomStatus(LiveStatus)`. Because `CustomStatus` is capped at **16 KB UTF-16** by Durable Functions, any workflow whose executor lifecycle events sum to more than ~16 KB hard-fails at the orchestrator level once that ceiling is crossed — there is no public hook to trim, suppress, or compact `AccumulatedEvents` from runtime code.

Observed in production usage on `Microsoft.Agents.AI.DurableTask` `1.5.0-preview.260507.1` with `Microsoft.Agents.AI.Hosting.AzureFunctions`.

## The mechanism (decompiled from 1.5.0-preview.260507.1)

`Microsoft.Agents.AI.DurableTask` ships an internal `SuperstepState` whose `AccumulatedEvents` list lives for the entire orchestration:

```csharp
// SuperstepState
public List AccumulatedEvents { get; } = new List();
public DurableWorkflowLiveStatus LiveStatus { get; } = new DurableWorkflowLiveStatus();
```

The orchestrator's per-executor loop appends each activity's serialized events:

```csharp
state.AccumulatedEvents.AddRange(executorResultInfo.Events);
flag |= executorResultInfo.HaltRequested;
if (!context.IsReplaying)
{
PublishEventsToLiveStatus(context, state);
}
```

`PublishEventsToLiveStatus` then projects the cumulative list directly into `CustomStatus`:

```csharp
private static void PublishEventsToLiveStatus(TaskOrchestrationContext context, SuperstepState state)
{
state.LiveStatus.Events = state.AccumulatedEvents;
context.SetCustomStatus((object)state.LiveStatus);
}
```

`executorResultInfo.Events` comes from each activity's `IWorkflowContext.OutboundEvents`, which carries `ExecutorInvokedEvent`, `ExecutorCompletedEvent`, and `WorkflowOutputEvent` entries — the `WorkflowOutputEvent` payload contains the executor's typed output. Across N executors, `AccumulatedEvents` grows by the serialized size of each.

## Repro

Any MAF workflow registered against the AzureFunctions companion host with:

- 6+ scoped executors chained sequentially
- typed outputs in the 1–4 KB serialized range (i.e. records with several fields plus a moderately sized `string` property, common for document-extraction / structured-response workflows)
- run via the Functions companion (Durable Functions backed)

Hits this at the orchestrator step that runs `PublishEventsToLiveStatus` after the cumulative size crosses 16 KB:

```
Microsoft.Azure.WebJobs.Extensions.DurableTask:
CustomStatus is too large: limit = 16 KB (UTF-16), actual = 17.32 KB.
```

Concrete repro available in [`connells-tech/ai-agent-runtime-dusty`](https://github.com/connells-tech/ai-agent-runtime-dusty) — the document-processing workflow ([`src/Dusty.WorkflowFunctions/DocumentProcessingFunctionsWorkflow.cs`](https://github.com/connells-tech/ai-agent-runtime-dusty/blob/main/src/Dusty.WorkflowFunctions/DocumentProcessingFunctionsWorkflow.cs)) chains 6 executors (entry-unpack → validate-inputs → load-configuration → extract-layout → analyse-document → validate-outputs). The orchestrator fails at superstep 7 with exactly the message above. Per-activity sizes are well-behaved; cumulative is not.

## Why per-activity workarounds don't fix it

`ConnellsGroup.Ai.Agent.Base` ships an opt-in `ScopedWorkflowExecutor` with `enableStatusSlimming: true` that clears `ExecutorInvokedEvent` and prior `ExecutorCompletedEvent` from `IWorkflowContext.OutboundEvents` after the application handler returns. This trims what `executorResultInfo.Events` will contain when MAF reads `OutboundEvents` post-return — but the orchestrator's `AccumulatedEvents` already holds events from *prior* supersteps, and those cannot be reached from inside an activity. Verified empirically: the same workflow fails at the same byte count (17.32 KB) with and without the slimming hook enabled.

Manual reflection workarounds against `OutboundEvents` have the same limitation — they can only affect the current activity's contribution. The orchestrator's `SuperstepState` is not exposed to executors.

## Proposed fixes (any one would unblock)

1. **Make `AccumulatedEvents` trim-able from a public hook.** Expose an option on `WorkflowOptions` (or equivalent) like `OnSuperstepCompleted: Func, IReadOnlyList>` that lets the host filter/compact the accumulated event log before `PublishEventsToLiveStatus` runs. Lowest-risk; orchestrators that don't opt in retain current behavior.

2. **Reset `AccumulatedEvents` per superstep.** Change `PublishEventsToLiveStatus` to publish only the current superstep's events, on the grounds that Durable history retains everything for after-the-fact retrieval, and `CustomStatus` is the live-status projection. Replace the cumulative semantic with a per-superstep one. Behavioral change but matches the way external streaming consumers typically poll (latest delta, not full history).

3. **Skip serializing executor lifecycle events into CustomStatus by default.** Most production consumers care about `LiveStatus.PendingEvents` (HITL request ports) and final results, not per-executor `ExecutorInvokedEvent`/`ExecutorCompletedEvent` payloads. An opt-in to omit lifecycle event payloads from `LiveStatus.Events` while keeping `PendingEvents` and result metadata would solve this for the common case.

Option 1 is the most flexible and the smallest behavioral change; option 3 is the easiest to recommend by default since the data is duplicated in history anyway.

## Workarounds we're using today

- `enableStatusSlimming: true` on `ScopedWorkflowExecutor` (helps for small workflows; insufficient here)
- Synchronous extraction path that bypasses the orchestrator entirely (works, but loses the durable-execution semantics this whole satellite exists to provide)
- Splitting workflows into smaller chained orchestrations (significant refactor)

## Impact

Any consumer building a non-trivial structured-output workflow on `Microsoft.Agents.AI.DurableTask` + Azure Functions will hit this once executor count × per-executor output size exceeds the ~16 KB envelope. The threshold is reached quickly for document processing, multi-step extraction, or any workflow with typed outputs larger than a few hundred bytes per executor.

## Versions

- `Microsoft.Agents.AI.DurableTask` 1.5.0-preview.260507.1
- `Microsoft.Agents.AI.Workflows` 1.5.0-preview.260507.1
- `Microsoft.Agents.AI.Hosting.AzureFunctions` 1.5.0-preview.260507.1
- `Microsoft.Azure.Functions.Worker` (Azure Functions Core Tools 4.9.0)
- `Microsoft.Azure.WebJobs.Extensions.DurableTask` 3.8.2

---

Filed via Claude Code on behalf of `@mfalkiewicz` (Connells Group) — happy to provide a minimal repro repo / additional reflection-dump if useful.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.