dapr / dapr/dotnet-sdk

Actor proxy reuses HTTP request when performing a retry to call an actor

Open
#1,264 1 comment 0 reactions 0 assignees View on GitHub
kind/bug
Dominant language
C#
Stars
1.2k
Forks
378
Avg merge
2d 1h
Merged PRs (30d)
6

Description

## Expected Behavior

When a failure occurs during actor processing and a retry policy is in place, the actor proxy should generate a new HTTP request for each retry.

## Actual Behavior

At the moment, the entire HTTP request is reused. This results in an exception with the following stack trace:

> ---> Dapr.Actors.ActorInvokeException: The request message was already sent. Cannot send the same request message multiple times.
--- End of inner exception stack trace ---
at Dapr.Actors.DaprHttpInteractor.InvokeActorMethodWithRemotingAsync(ActorMessageSerializersManager serializersManager, IActorRequestMessage remotingRequestRequestMessage, CancellationToken cancellationToken)
at Dapr.Actors.Communication.Client.ActorRemotingClient.InvokeAsync(IActorRequestMessage remotingRequestMessage, CancellationToken cancellationToken)
at Dapr.Actors.Client.ActorProxy.InvokeMethodAsync(Int32 interfaceId, Int32 methodId, String methodName, IActorRequestMessageBody requestMsgBodyValue, CancellationToken cancellationToken)

## Steps to Reproduce the Problem

1. Build an API controller and bind it to a Pub/Sub topic of your choice that has a try/catch around the controller method (or global exception handling middleware).
2. In your controller, resolve a proxy to an actor of some kind, and pass the contents of the CloudEvent to your actor.
3. Build your actor to call some HTTP endpoint that will return a 400 of any kind.
4. Observe that when the 400 is returned to the actor, the Dapr actor exception swallows the 400 and returns a doubly-wrapped actor exception which does not contain the original error from the HTTP endpoint as the inner-most exception.
5. In your controller/middleware, when the actor exception is thrown, you are unable deduce the original exception due to the nature of Dapr actor exceptions, so you return a 500.
6. Dapr Pub/Sub resiliency will trigger a re-queue/re-process of the event, rather than nack'ing the event off the queue.
7. You will now observe an exception being thrown in the Dapr communications SDK stating that the HTTP request has already been sent as per the stack trace above.

## Release Note

RELEASE NOTE:

Contributor guide

Open the contributing guide

Research direction

Start at DaprHttpInteractor.InvokeActorMethodWithRemotingAsync and trace its calls through ActorRemotingClient.InvokeAsync and ActorProxy.InvokeMethodAsync. Reproduce the retry path described in the issue and verify that each retry uses a new HTTP request rather than a previously sent one, while checking the reported exception behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
csharp
Domain
api, backend
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.