Unable to run workflows in kubernetes
- Dominant language
- C#
- Stars
- 1.2k
- Forks
- 378
- Avg merge
- 2d 1h
- Merged PRs (30d)
- 6
Description
## Expected Behavior
Being able to run workflows normally on kubernetes the same way it works locally
## Actual Behavior
Somehow workflow does not start and times out as shown in the log:
```
time="2023-03-23T12:19:56.31185114Z" level=debug msg="invoking method 'CreateWorkflowInstance' on workflow actor 'asdf'" app_id=wfapp instance=wfapp-7cb4ccd64b-fgn68 scope=dapr.runtime.wfengine type=log ver=1.10.4
time="2023-03-23T12:19:56.311875005Z" level=debug msg="asdf: loading workflow state" app_id=wfapp instance=wfapp-7cb4ccd64b-fgn68 scope=dapr.runtime.wfengine type=log ver=1.10.4
time="2023-03-23T12:19:56.347261672Z" level=debug msg="asdf: creating 'start-4ac7fe35' reminder with DueTime = 0s" app_id=wfapp instance=wfapp-7cb4ccd64b-fgn68 scope=dapr.runtime.wfengine type=log ver=1.10.4
time="2023-03-23T12:19:56.370525261Z" level=debug msg="starting to read reminders for actor type dapr.internal.wfengine.workflow (migrate=false), with metadata id 00000000-0000-0000-0000-000000000000 and 0 partitions" app_id=wfapp instance=wfapp-7cb4ccd64b-fgn68 scope=dapr.runtime.actor type=log ver=1.10.4
time="2023-03-23T12:19:56.377679829Z" level=debug msg="read reminders from actors||dapr.internal.wfengine.workflow without partition: " app_id=wfapp instance=wfapp-7cb4ccd64b-fgn68 scope=dapr.runtime.actor type=log ver=1.10.4
time="2023-03-23T12:19:56.37770212Z" level=debug msg="finished reading reminders for actor type dapr.internal.wfengine.workflow (migrate=false), with metadata id 00000000-0000-0000-0000-000000000000 and no partitions: total of 0 reminders" app_id=wfapp instance=wfapp-7cb4ccd64b-fgn68 scope=dapr.runtime.actor type=log ver=1.10.4
time="2023-03-23T12:19:56.377719283Z" level=debug msg="saving 1 reminders in actors||dapr.internal.wfengine.workflow ..." app_id=wfapp instance=wfapp-7cb4ccd64b-fgn68 scope=dapr.runtime.actor type=log ver=1.10.4
time="2023-03-23T12:19:56.403174193Z" level=debug msg="asdf: saving 3 keys to actor state store" app_id=wfapp instance=wfapp-7cb4ccd64b-fgn68 scope=dapr.runtime.wfengine type=log ver=1.10.4
time="2023-03-23T12:19:56.40325296Z" level=debug msg="#operations=3,partitionkey=wfapp||dapr.internal.wfengine.workflow||asdf" app_id=wfapp component="actor-state-store (state.azure.cosmosdb/v1)" instance=wfapp-7cb4ccd64b-fgn68 scope=dapr.contrib type=log ver=1.10.4
time="2023-03-23T12:19:56.403305396Z" level=debug msg="executing reminder start-4ac7fe35 for actor type dapr.internal.wfengine.workflow with id asdf" app_id=wfapp instance=wfapp-7cb4ccd64b-fgn68 scope=dapr.runtime.actor type=log ver=1.10.4
time="2023-03-23T12:19:56.413475002Z" level=debug msg="Operation 0 completed with status code 201" app_id=wfapp component="actor-state-store (state.azure.cosmosdb/v1)" instance=wfapp-7cb4ccd64b-fgn68 scope=dapr.contrib type=log ver=1.10.4
time="2023-03-23T12:19:56.413498607Z" level=debug msg="Operation 1 completed with status code 201" app_id=wfapp component="actor-state-store (state.azure.cosmosdb/v1)" instance=wfapp-7cb4ccd64b-fgn68 scope=dapr.contrib type=log ver=1.10.4
time="2023-03-23T12:19:56.413514687Z" level=debug msg="Operation 2 completed with status code 201" app_id=wfapp component="actor-state-store (state.azure.cosmosdb/v1)" instance=wfapp-7cb4ccd64b-fgn68 scope=dapr.contrib type=log ver=1.10.4
time="2023-03-23T12:19:56.413558719Z" level=info msg="created new workflow instance with ID 'asdf'" app_id=wfapp component="dapr (workflow.dapr/v1)" instance=wfapp-7cb4ccd64b-fgn68 scope=dapr.contrib type=log ver=1.10.4
time="2023-03-23T12:19:56.413586441Z" level=debug msg="&{asdf}" app_id=wfapp instance=wfapp-7cb4ccd64b-fgn68 scope=dapr.runtime.http type=log ver=1.10.4
time="2023-03-23T12:19:56.413585635Z" level=debug msg="invoking reminder 'start-4ac7fe35' on workflow actor 'asdf'" app_id=wfapp instance=wfapp-7cb4ccd64b-fgn68 scope=dapr.runtime.wfengine type=log ver=1.10.4
time="2023-03-23T12:19:56.413677475Z" level=debug msg="orchestration-processor: processing work item: asdf (1 event(s))" app_id=wfapp instance=wfapp-7cb4ccd64b-fgn68 scope=dapr.runtime.wfengine type=log ver=1.10.4
time="2023-03-23T12:19:56.413723611Z" level=debug msg="asdf: received work item with 1 new event(s): [ExecutionStarted]" app_id=wfapp instance=wfapp-7cb4ccd64b-fgn68 scope=dapr.runtime.wfengine type=log ver=1.10.4
time="2023-03-23T12:19:56.413745232Z" level=debug msg="asdf: got orchestration runtime state: name=OrderProcessingWorkflow, status=PENDING, events=0, age=(new)" app_id=wfapp instance=wfapp-7cb4ccd64b-fgn68 scope=dapr.runtime.wfengine type=log ver=1.10.4
time="2023-03-23T12:19:56.413768135Z" level=info msg="asdf: starting new 'OrderProcessingWorkflow' instance." app_id=wfapp instance=wfapp-7cb4ccd64b-fgn68 scope=dapr.runtime.wfengine type=log ver=1.10.4
time="2023-03-23T12:19:56.413786479Z" level=debug msg="asdf: invoking orchestrator" app_id=wfapp instance=wfapp-7cb4ccd64b-fgn68 scope=dapr.runtime.wfengine type=log ver=1.10.4
time="2023-03-23T12:20:26.414052507Z" level=warning msg="asdf: execution timed-out and will be retried later" app_id=wfapp instance=wfapp-7cb4ccd64b-fgn68 scope=dapr.runtime.wfengine type=log ver=1.10.4
```
## Steps to Reproduce the Problem
It should be reproducible, but I don't have the exact steps for now. I used azure kubernetes service. Make sure to call `AddDaprWorkflow()` which will add a dapr client with `localhost` endpoint. (The default value when calling `AddDaprClient()` is `127.0.0.1`)
## Workaround
I wrote my own `AddDaprWorkflow()` method with a `TryGetGrpcAddress()` that returns an endpoint with `127.0.0.1`. Call `AddWorkflowsAndActivitiesToRegistry()` via reflection
```
var dynMethod = typeof(WorkflowRuntimeOptions).GetMethod(
"AddWorkflowsAndActivitiesToRegistry",
BindingFlags.NonPublic | BindingFlags.Instance
);
dynMethod.Invoke(options, new object[] { registry });
```
It then works
## Related issues
- https://github.com/dapr/dotnet-sdk/issues/1032
introduces `DAPR_HOST` which makes it easier to override host by `127.0.0.1`. The associated PR also changes the default host to `localhost` which might hinder running workflows on kubernetes by default.
- Looking forward to https://github.com/dapr/dotnet-sdk/issues/1061
Contributor guide
Research direction
Start with AddDaprWorkflow and the default endpoint behavior of AddDaprClient, then inspect WorkflowRuntimeOptions, TryGetGrpcAddress, and AddWorkflowsAndActivitiesToRegistry. Reproduce the timeout on Azure Kubernetes Service using the logged workflow setup and compare it with the 127.0.0.1 workaround. Done means workflows start normally in Kubernetes without reflection or a custom AddDaprWorkflow method.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- csharp, kubernetes
- Domain
- backend, distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100