Bot never joins Team Meeting: Server Internal Error. DiagCode: 500#1203003.@

Describe the issue

When inviting the bot to a Teams call we receive a HTTP 500 error and the bot never joins the call.

Code Snippet

var statefulCall = await this.Client.Calls().AddAsync(joinParams).ConfigureAwait(false);

This runs successfully, but we never receive the established callback.

Expected behavior

We expect to receive a series of callbacks from Microsoft. We expect to receive the established callback that eventually resolves to the Bot joining a Teams call.

Graph SDK (please complete the following information):

Microsoft.Graph.Communications.Calls" Version="1.2.0.850"
Microsoft.Graph.Communications.Client" Version="1.2.0.850"
Microsoft.Graph.Communications.Common" Version="1.2.0.850"
Microsoft.Graph.Communications.Core" Version="1.2.0.850"
Microsoft.Graph.Communications.Calls.Media" Version="1.2.0.850"
Microsoft.Skype.Bots.Media" Version="1.27.0.2-alpha"

Call ID

Time: 2024-10-14 16:13:09.754 | time="2024-10-14T15:13:09.7547304Z
callChainId: ceb776d1-051b-4257-b170-4f53376418e7

Logs

Initial POST api/calling request where state is establishing

{
  "@odata.type": "#microsoft.graph.commsNotifications",
  "value": [
    {
      "@odata.type": "#microsoft.graph.commsNotification",
      "changeType": "updated",
      "resource": "/app/calls/00003680-2f08-421e-8420-b8684632e4c5",
      "resourceUrl": "/communications/calls/00003680-2f08-421e-8420-b8684632e4c5",
      "resourceData": {
        "@odata.type": "#microsoft.graph.call",
        "state": "establishing",
        "chatInfo": {
          "@odata.type": "#microsoft.graph.chatInfo",
          "threadId": "19:meeting_MzdhZDk5YTctMTNlMi00ZGYzLWE3NjAtMmQ4MzlhY2RmZmYy@thread.v2",
          "messageId": "0"
        },
        "meetingInfo": {
          "@odata.type": "#microsoft.graph.organizerMeetingInfo",
          "organizer": {
            "@odata.type": "#microsoft.graph.identitySet",
            "user": {
              "@odata.type": "#microsoft.graph.identity",
              "id": "f353f265-2c71-41c1-9537-9df59b7f2965",
              "tenantId": "<ommitted>"
            }
          },
          "callChainId": "d8bcb98a-6906-4124-baf2-f952d158a8d5"
        }
      }
    }
  ]
}

After a further 4 establishing requests we receive this as the 6th request that updates state to terminated:

{
  "@odata.type": "#microsoft.graph.commsNotifications",
  "value": [
    {
      "@odata.type": "#microsoft.graph.commsNotification",
      "changeType": "deleted",
      "resource": "/app/calls/0a003680-3adb-42a1-ac37-8cd856289569",
      "resourceUrl": "/communications/calls/0a003680-3adb-42a1-ac37-8cd856289569",
      "resourceData": {
        "@odata.type": "#microsoft.graph.call",
        "state": "terminated",
        "resultInfo": {
          "@odata.type": "#microsoft.graph.resultInfo",
          "code": 500,
          "subcode": 1203003,
          "message": "Server Internal Error. DiagCode: 500#1203003.@"
        },
        "chatInfo": {
          "@odata.type": "#microsoft.graph.chatInfo",
          "threadId": "19:meeting_MzdhZDk5YTctMTNlMi00ZGYzLWE3NjAtMmQ4MzlhY2RmZmYy@thread.v2",
          "messageId": "0"
        },
        "meetingInfo": {
          "@odata.type": "#microsoft.graph.organizerMeetingInfo",
          "organizer": {
            "@odata.type": "#microsoft.graph.identitySet",
            "user": {
              "@odata.type": "#microsoft.graph.identity",
              "id": "f353f265-2c71-41c1-9537-9df59b7f2965",
              "tenantId": "<ommitted>"
            }
          },
          "callChainId": "ceb776d1-051b-4257-b170-4f53376418e7"
        }
      }
    }
  ]
}

Additional context

• We have already setup the bot to run as a single instance, with this we're trying to run multiple instances. The implementation of the k8s infra takes inspiration from this setup.
• We're running this on AWS, we'll soon be moving to Azure as was recommended to my colleague a little while ago here
• The Bot has never joined the call with this setup

No repository file or test is named. Start at the Calls().AddAsync entry point and the notification handling for establishing and terminated states, then review the Kubernetes deployment context and SDK versions listed in the issue. Done means identifying the cause of the 500#1203003 termination and receiving the established callback so the bot joins the Teams call.