dapr / dapr/dotnet-sdk

Unable to run workflows in kubernetes

Open
#1,070 2 comments 0 reactions 0 assignees View on GitHub
kind/bug
Dominant language
C#
Stars
1.2k
Forks
378
Avg merge
2d 1h
Merged PRs (30d)
6

Description

## Expected Behavior

Being able to run workflows normally on kubernetes the same way it works locally

## Actual Behavior

Somehow workflow does not start and times out as shown in the log:
```
time="2023-03-23T12:19:56.31185114Z" level=debug msg="invoking method 'CreateWorkflowInstance' on workflow actor 'asdf'" app_id=wfapp instance=wfapp-7cb4ccd64b-fgn68 scope=dapr.runtime.wfengine type=log ver=1.10.4
time="2023-03-23T12:19:56.311875005Z" level=debug msg="asdf: loading workflow state" app_id=wfapp instance=wfapp-7cb4ccd64b-fgn68 scope=dapr.runtime.wfengine type=log ver=1.10.4
time="2023-03-23T12:19:56.347261672Z" level=debug msg="asdf: creating 'start-4ac7fe35' reminder with DueTime = 0s" app_id=wfapp instance=wfapp-7cb4ccd64b-fgn68 scope=dapr.runtime.wfengine type=log ver=1.10.4
time="2023-03-23T12:19:56.370525261Z" level=debug msg="starting to read reminders for actor type dapr.internal.wfengine.workflow (migrate=false), with metadata id 00000000-0000-0000-0000-000000000000 and 0 partitions" app_id=wfapp instance=wfapp-7cb4ccd64b-fgn68 scope=dapr.runtime.actor type=log ver=1.10.4
time="2023-03-23T12:19:56.377679829Z" level=debug msg="read reminders from actors||dapr.internal.wfengine.workflow without partition: " app_id=wfapp instance=wfapp-7cb4ccd64b-fgn68 scope=dapr.runtime.actor type=log ver=1.10.4
time="2023-03-23T12:19:56.37770212Z" level=debug msg="finished reading reminders for actor type dapr.internal.wfengine.workflow (migrate=false), with metadata id 00000000-0000-0000-0000-000000000000 and no partitions: total of 0 reminders" app_id=wfapp instance=wfapp-7cb4ccd64b-fgn68 scope=dapr.runtime.actor type=log ver=1.10.4
time="2023-03-23T12:19:56.377719283Z" level=debug msg="saving 1 reminders in actors||dapr.internal.wfengine.workflow ..." app_id=wfapp instance=wfapp-7cb4ccd64b-fgn68 scope=dapr.runtime.actor type=log ver=1.10.4
time="2023-03-23T12:19:56.403174193Z" level=debug msg="asdf: saving 3 keys to actor state store" app_id=wfapp instance=wfapp-7cb4ccd64b-fgn68 scope=dapr.runtime.wfengine type=log ver=1.10.4
time="2023-03-23T12:19:56.40325296Z" level=debug msg="#operations=3,partitionkey=wfapp||dapr.internal.wfengine.workflow||asdf" app_id=wfapp component="actor-state-store (state.azure.cosmosdb/v1)" instance=wfapp-7cb4ccd64b-fgn68 scope=dapr.contrib type=log ver=1.10.4
time="2023-03-23T12:19:56.403305396Z" level=debug msg="executing reminder start-4ac7fe35 for actor type dapr.internal.wfengine.workflow with id asdf" app_id=wfapp instance=wfapp-7cb4ccd64b-fgn68 scope=dapr.runtime.actor type=log ver=1.10.4
time="2023-03-23T12:19:56.413475002Z" level=debug msg="Operation 0 completed with status code 201" app_id=wfapp component="actor-state-store (state.azure.cosmosdb/v1)" instance=wfapp-7cb4ccd64b-fgn68 scope=dapr.contrib type=log ver=1.10.4
time="2023-03-23T12:19:56.413498607Z" level=debug msg="Operation 1 completed with status code 201" app_id=wfapp component="actor-state-store (state.azure.cosmosdb/v1)" instance=wfapp-7cb4ccd64b-fgn68 scope=dapr.contrib type=log ver=1.10.4
time="2023-03-23T12:19:56.413514687Z" level=debug msg="Operation 2 completed with status code 201" app_id=wfapp component="actor-state-store (state.azure.cosmosdb/v1)" instance=wfapp-7cb4ccd64b-fgn68 scope=dapr.contrib type=log ver=1.10.4
time="2023-03-23T12:19:56.413558719Z" level=info msg="created new workflow instance with ID 'asdf'" app_id=wfapp component="dapr (workflow.dapr/v1)" instance=wfapp-7cb4ccd64b-fgn68 scope=dapr.contrib type=log ver=1.10.4
time="2023-03-23T12:19:56.413586441Z" level=debug msg="&{asdf}" app_id=wfapp instance=wfapp-7cb4ccd64b-fgn68 scope=dapr.runtime.http type=log ver=1.10.4
time="2023-03-23T12:19:56.413585635Z" level=debug msg="invoking reminder 'start-4ac7fe35' on workflow actor 'asdf'" app_id=wfapp instance=wfapp-7cb4ccd64b-fgn68 scope=dapr.runtime.wfengine type=log ver=1.10.4
time="2023-03-23T12:19:56.413677475Z" level=debug msg="orchestration-processor: processing work item: asdf (1 event(s))" app_id=wfapp instance=wfapp-7cb4ccd64b-fgn68 scope=dapr.runtime.wfengine type=log ver=1.10.4
time="2023-03-23T12:19:56.413723611Z" level=debug msg="asdf: received work item with 1 new event(s): [ExecutionStarted]" app_id=wfapp instance=wfapp-7cb4ccd64b-fgn68 scope=dapr.runtime.wfengine type=log ver=1.10.4
time="2023-03-23T12:19:56.413745232Z" level=debug msg="asdf: got orchestration runtime state: name=OrderProcessingWorkflow, status=PENDING, events=0, age=(new)" app_id=wfapp instance=wfapp-7cb4ccd64b-fgn68 scope=dapr.runtime.wfengine type=log ver=1.10.4
time="2023-03-23T12:19:56.413768135Z" level=info msg="asdf: starting new 'OrderProcessingWorkflow' instance." app_id=wfapp instance=wfapp-7cb4ccd64b-fgn68 scope=dapr.runtime.wfengine type=log ver=1.10.4
time="2023-03-23T12:19:56.413786479Z" level=debug msg="asdf: invoking orchestrator" app_id=wfapp instance=wfapp-7cb4ccd64b-fgn68 scope=dapr.runtime.wfengine type=log ver=1.10.4
time="2023-03-23T12:20:26.414052507Z" level=warning msg="asdf: execution timed-out and will be retried later" app_id=wfapp instance=wfapp-7cb4ccd64b-fgn68 scope=dapr.runtime.wfengine type=log ver=1.10.4
```

## Steps to Reproduce the Problem

It should be reproducible, but I don't have the exact steps for now. I used azure kubernetes service. Make sure to call `AddDaprWorkflow()` which will add a dapr client with `localhost` endpoint. (The default value when calling `AddDaprClient()` is `127.0.0.1`)

## Workaround
I wrote my own `AddDaprWorkflow()` method with a `TryGetGrpcAddress()` that returns an endpoint with `127.0.0.1`. Call `AddWorkflowsAndActivitiesToRegistry()` via reflection
```
var dynMethod = typeof(WorkflowRuntimeOptions).GetMethod(
"AddWorkflowsAndActivitiesToRegistry",
BindingFlags.NonPublic | BindingFlags.Instance
);
dynMethod.Invoke(options, new object[] { registry });
```
It then works

## Related issues
- https://github.com/dapr/dotnet-sdk/issues/1032
introduces `DAPR_HOST` which makes it easier to override host by `127.0.0.1`. The associated PR also changes the default host to `localhost` which might hinder running workflows on kubernetes by default.
- Looking forward to https://github.com/dapr/dotnet-sdk/issues/1061

Contributor guide

Open the contributing guide

Research direction

Start with AddDaprWorkflow and the default endpoint behavior of AddDaprClient, then inspect WorkflowRuntimeOptions, TryGetGrpcAddress, and AddWorkflowsAndActivitiesToRegistry. Reproduce the timeout on Azure Kubernetes Service using the logged workflow setup and compare it with the 127.0.0.1 workaround. Done means workflows start normally in Kubernetes without reflection or a custom AddDaprWorkflow method.

Written by the indexing model from the issue text.

Assessment

Tech stack
csharp, kubernetes
Domain
backend, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.