microsoft / microsoft/durabletask-netherite

Flex Consumption Plan: The specified taskhub does not exist (TaskHub=XXX, StorageConnectionName=AzureWebJobsStorage.")

Open
#421 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

P2
Dominant language
C#
Stars
239
Forks
37
Avg merge
1d 10h
Merged PRs (30d)
2

Description

I have created and deployed Durable Function App that uses Netherite, but can't start an orchestration.

Error:

Status(StatusCode="Internal", Detail="Failed to start instance with ID c2f8ebaa996847f7816e480d0c63464e.
Inner Exception message: The specified taskhub does not exist (TaskHub=XXX, StorageConnectionName=AzureWebJobsStorage.")

StackTrace:

[
    {
        "severityLevel": "Error",
        "outerId": "0",
        "message": "Status(StatusCode=\"Internal\", Detail=\"Failed to start instance with ID af81b4b4895b4304b283bcb99559bd69.\nInner Exception message: The specified taskhub does not exist (TaskHub=XXX, StorageConnectionName=AzureWebJobsStorage.\")",
        "type": "Grpc.Core.RpcException",
        "id": "15915024",
        "parsedStack": [
            {
                "assembly": "System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e",
                "method": "System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw",
                "level": 0,
                "line": 0
            },
            {
                "assembly": "System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e",
                "method": "System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess",
                "level": 1,
                "line": 0
            },
            {
                "assembly": "System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e",
                "method": "System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification",
                "level": 2,
                "line": 0
            },
            {
                "assembly": "System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e",
                "method": "System.Runtime.CompilerServices.TaskAwaiter`1.GetResult",
                "level": 3,
                "line": 0
            },
            {
                "assembly": "Microsoft.DurableTask.Client.Grpc, Version=1.2.4.0, Culture=neutral, PublicKeyToken=6a4c0315c2d1d937",
                "method": "Microsoft.DurableTask.Client.Grpc.GrpcDurableTaskClient+<ScheduleNewOrchestrationInstanceAsync>d__12.MoveNext",
                "level": 4,
                "line": 0
            },
            {
                "assembly": "System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e",
                "method": "System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw",
                "level": 5,
                "line": 0
            },
            {
                "assembly": "System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e",
                "method": "System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess",
                "level": 6,
                "line": 0
            },
            {
                "assembly": "System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e",
                "method": "System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification",
                "level": 7,
                "line": 0
            },
            {
                "assembly": "System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e",
                "method": "System.Runtime.CompilerServices.TaskAwaiter`1.GetResult",
                "level": 8,
                "line": 0
            },
            {
                "assembly": "YYY, Version=2.3.0.13281, Culture=neutral, PublicKeyToken=null",
                "method": "YYY.HttpStartFunction+<Run>d__4.MoveNext",
                "level": 9,
                "line": 64,
                "fileName": "D:\\a\\1\\s\\src\\YYY\\HttpStartFunction.cs"
            }
        ]
    }
]

Netherite exception:

Failed to start instanceId c2f8ebaa996847f7816e480d0c63464e due to internal exception.
Exception trace: DurableTask.Netherite.NetheriteConfigurationException: The specified taskhub does not exist (TaskHub=XXX, StorageConnectionName=AzureWebJobsStorage
at DurableTask.Netherite.Faster.FasterStorageLayer.DurableTask.Netherite.IStorageLayer.TryLoadTaskhubAsync(Boolean throwIfNotFound) in /_/src/DurableTask.Netherite/StorageLayer/Faster/FasterStorageProvider.cs:line 116
at DurableTask.Netherite.EventHubsTransport.EventHubsTransport.DurableTask.Netherite.ITransportLayer.StartAsync() in /_/src/DurableTask.Netherite/TransportLayer/EventHubs/EventHubsTransport.cs:line 93
at DurableTask.Netherite.NetheriteOrchestrationService.StartClientAsync() in /_/src/DurableTask.Netherite/OrchestrationService/NetheriteOrchestrationService.cs:line 415
at DurableTask.Netherite.NetheriteOrchestrationService.<>c__DisplayClass92_0`1.<<TryTransition>g__conditionalTransition|0>d.MoveNext() in /_/src/DurableTask.Netherite/OrchestrationService/NetheriteOrchestrationService.cs:line 387
--- End of stack trace from previous location ---
at DurableTask.Netherite.NetheriteOrchestrationService.TryStartAsync(Boolean clientOnly) in /_/src/DurableTask.Netherite/OrchestrationService/NetheriteOrchestrationService.cs:line 344
at DurableTask.Netherite.NetheriteOrchestrationService.GetClientAsync() in /_/src/DurableTask.Netherite/OrchestrationService/NetheriteOrchestrationService.cs:line 74
at DurableTask.Netherite.NetheriteOrchestrationService.DurableTask.Core.IOrchestrationServiceClient.CreateTaskOrchestrationAsync(TaskMessage creationMessage, OrchestrationStatus[] dedupeStatuses) in /_/src/DurableTask.Netherite/OrchestrationService/NetheriteOrchestrationService.cs:line 656
at DurableTask.Core.TaskHubClient.InternalCreateOrchestrationInstanceWithRaisedEventAsync(String orchestrationName, String orchestrationVersion, String orchestrationInstanceId, Object orchestrationInput, IDictionary`2 orchestrationTags, OrchestrationStatus[] dedupeStatuses, String eventName, Object eventData, Nullable`1 startAt) in /_/src/DurableTask.Core/TaskHubClient.cs:line 645
at Microsoft.Azure.WebJobs.Extensions.DurableTask.DurableClient.Microsoft.Azure.WebJobs.Extensions.DurableTask.IDurableOrchestrationClient.StartNewAsync[T](String orchestratorFunctionName, String instanceId, T input) in D:\a\_work\1\s\src\WebJobs.Extensions.DurableTask\ContextImplementations\DurableClient.cs:line 215
at Microsoft.Azure.WebJobs.Extensions.DurableTask.LocalGrpcListener.TaskHubGrpcServer.StartInstance(CreateInstanceRequest request, ServerCallContext context) in D:\a\_work\1\s\src\WebJobs.Extensions.DurableTask\LocalGrpcListener.cs:line 158.. InstanceId: c2f8ebaa996847f7816e480d0c63464e. Function: MainOrchestrator. HubName: XXX. AppName: XXX. SlotName: Production. ExtensionVersion: 2.13.5. SequenceNumber: 2.

Deployed using AzureFunctionApp@2 - Azure Functions Deploy v2 task

- task: AzureFunctionApp@2
  displayName: 'Deploy to Flex Consumption Function App'
  inputs:
    azureSubscription: '$(azureSubscription)'
    appType: 'functionAppLinux' 
    isFlexConsumption: true
    appName: '$(appName)'
    package: '$(Build.ArtifactStagingDirectory)/$(buildVersion).zip'
    #appSettings: # string. App settings. 
    deploymentMethod: 'auto'

Function App is showing the list of functions.
image

Function app contains all the required app settings,

[
  {
    "name": "AzureFunctionsJobHost__extensions__durableTask__storageProvider__eventHubsConnectionName",
    "value": "EventHubsConnection",
    "slotSetting": false
  },
  {
    "name": "AzureFunctionsJobHost__extensions__durableTask__storageProvider__LoadInformationAzureTableName",
    "value": "DurableTaskPartitions",
    "slotSetting": false
  },
  {
    "name": "AzureFunctionsJobHost__extensions__durableTask__storageProvider__PartitionCount",
    "value": "12",
    "slotSetting": false
  },
  {
    "name": "AzureFunctionsJobHost__extensions__durableTask__storageProvider__storageConnectionName",
    "value": "AzureWebJobsStorage",
    "slotSetting": false
  },
  {
    "name": "AzureFunctionsJobHost__extensions__durableTask__storageProvider__type",
    "value": "Netherite",
    "slotSetting": false
  },
  {
    "name": "AzureWebJobsStorage",
    "value": "<AzureWebJobsStorage>",
    "slotSetting": false
  },
  {
    "name": "DEPLOYMENT_STORAGE_CONNECTION_STRING",
    "value": "<DEPLOYMENT_STORAGE_CONNECTION_STRING>",
    "slotSetting": false
  },
  {
    "name": "EventHubsConnection",
    "value": "<EventHubsConnection>",
    "slotSetting": false
  },
  {
    "name": "FUNCTIONS_EXTENSION_VERSION",
    "value": "~4",
    "slotSetting": false
  }
]

In the Storage Account container for Task Hub and DurableTaskPartitions is missing:
image

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reviewing the configuration and the stack-trace locations in FasterStorageProvider.cs, EventHubsTransport.cs, and NetheriteOrchestrationService.cs. Reproduce the Flex Consumption deployment with the shown app settings and determine why the task hub is not created or found; done means the task hub and DurableTaskPartitions storage exist and the orchestration starts successfully.

Written by the indexing model from the issue text.

Assessment

Tech stack
azure, csharp
Domain
backend, cloud, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.