Azure / Azure/azure-functions-host

RPC exception sanitizer corrupts Durable out-of-proc replay state for credential-like errors

Open
#11,896 0 comments 0 reactions 0 assignees View on GitHub
area: out-of-proc bug feature:durable functions Needs: Triage (Functions) regression
Dominant language
C#
Stars
2k
Forks
482
Avg merge
2d 12h
Merged PRs (30d)
38

Description

#### Check for a solution in the Azure portal

Reproduced locally; this is not production-specific.

#### Investigative information

- Timestamp: 2026-07-28 22:21 UTC
- Function App version: Azure Functions Runtime 4.1048.200.26180 / Core Tools 4.12.1
- Function App name: local minimal repro
- Function names: `orchestrator`, `activity`
- Durable extension: 3.8.0
- Python: 3.13.11
- `azure-functions-durable`: current `dev` at Azure/azure-functions-durable-python commit `8eae23d`

This also remains present in the current Functions host `dev` implementation.

#### Repro steps

1. Create a Python Durable orchestration that calls one activity.
2. Have the activity throw an exception whose message contains a credential token recognized by the Functions host sanitizer. The minimal trigger is `?sig=x` (the original report used a Blob SAS URL).

```python
@app.orchestration_trigger(context_name="context")
def orchestrator(context: df.DurableOrchestrationContext):
yield context.call_activity("activity")

@app.activity_trigger(input_name="input_data")
def activity(input_data):
raise RuntimeError("?sig=x")
```

3. Start the orchestration and query its status.
4. As a control, replace `?sig=x` with `x` and start a new orchestration.

Additional confirmed triggers include `&sig=`, `Token=`, `AccountKey=`, `SharedAccessKey=`, `key=`, and `?code=`. The signature value and the characters `/`, `+`, and `=` are not relevant.

#### Expected behavior

The orchestration reaches `Failed` status with the activity exception in its output, as it does for the `x` control.

#### Actual behavior

The orchestration reaches `Failed` with a misleading error:

```
Non-Deterministic workflow detected: A previous execution of this orchestration scheduled an activity task with sequence ID 0 ... but the current replay execution hasn't (yet?) scheduled this task.
```

#### Root cause

This is caused by the Functions host sanitizing the RPC exception's runtime message, not only its log representation:

- `RpcException` calls `Sanitizer.Sanitize(message)` for both `Exception.Message` and `RemoteMessage`: https://github.com/Azure/azure-functions-host/blob/a6c66fabd5ac3a3d457c380b53547fe5d318e072/src/WebJobs.Script/Workers/Rpc/RpcException.cs#L11-L17
- The sanitizer recognizes tokens such as `&sig=`, `?sig=`, `Token=`, and `key=` and replaces through the next quote/terminator: https://github.com/Azure/azure-functions-host/blob/a6c66fabd5ac3a3d457c380b53547fe5d318e072/src/WebJobs.Script/Sanitizer.cs#L18-L87
- Python Durable orchestrator failures carry replay state after `\n\n$OutOfProcData$:`: https://github.com/Azure/azure-functions-durable-python/blob/8eae23d3dc126fb3b6c9b708a58f94e0c5ba8f21/azure/durable_functions/models/TaskOrchestrationExecutor.py#L354-L361
- The Durable extension extracts that state from the RPC exception so it can replay already-scheduled actions before failing the orchestration: https://github.com/Azure/azure-functions-durable-extension/blob/c07ebda7f91d90cc3baf4916fca76191a841b4fe/src/WebJobs.Extensions.DurableTask/OutOfProcExceptionHelpers.cs#L48-L75

Sanitization mutates the embedded JSON replay-state envelope. In nested exception details, its next-quote replacement can remove a JSON escape and make the state invalid. The Durable extension can no longer replay the action list; DurableTask then reports the secondary non-determinism error.

The behavior was introduced by #10443 (`Sanitize exception logs`), which changed the transport-level `RpcException` in addition to log output. Current `dev` still applies the sanitizer there. Because `$OutOfProcData$` is also used by the JavaScript and PowerShell Durable SDKs, the underlying issue may affect other out-of-process languages.

The fix should preserve the raw RPC exception for runtime/control-channel consumers and sanitize only at logging sinks. If transport-level sanitization must remain, it needs to recognize and preserve the Durable out-of-proc state envelope as valid JSON.

#### Known workarounds

Do not include credentials or SAS query strings in exception messages. Strip the URL query string before raising/logging the exception. This is also preferable security practice.

#### Related information

- Original report and minimal repro: https://github.com/Azure/azure-functions-durable-python/issues/600
- Regression PR: https://github.com/Azure/azure-functions-host/pull/10443
- Programming language: Python, with likely cross-language impact
- Binding: Durable orchestration/activity triggers

Contributor guide

Open the contributing guide

Research direction

Start with src/WebJobs.Script/Workers/Rpc/RpcException.cs and src/WebJobs.Script/Sanitizer.cs, then read OutOfProcExceptionHelpers.cs and the linked Durable Python replay-state code. Run the minimal Python Durable orchestration with ?sig=x and the x control case. Done means the credential-like message no longer corrupts the embedded replay state and the orchestration fails with the activity exception rather than a non-determinism error.

Written by the indexing model from the issue text.

Assessment

Tech stack
csharp, python
Domain
backend, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
52/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.