TeamsRecordingBot not joining call and/or receiving callbacks on repeat deployment

I have a working deployment of the recording bot running in an AKS cluster and now am hitting problems trying to deploy it again on a bumped version and updated cert-manager and ingress. Certificates are issued fine and the pod deploys and receives the JoinCall POST request.

For some reason, despite me giving the bot all required permissions and setting up the callback address analogously to the already working bot, however, it is unable to join any calls. The main Graph POST call to /communication/calls seems to not succeed because I constantly get error 8522 when trying to keepAlive it after. I also tried to make the call manually using MSAL, and it returns "establishing" but the bot does not join in the end.

In the logs, I can see the following from the calls to the endpoint /communications/calls/CALLID/keepAlive:

"StatefulCall" pid:3240 tid:127 (GraphAuthClient.cs:117 SendHttpRequestAsync):
{
"error": {
"code": "8522",
"message": "Call not found."

One of the (many) Call IDs where this happened is: 5b1f6000-3163-457d-adb9-c8246802bbff.

The Bot joins normally otherwise and all related CallHandler, MediaStream etc. objects are created, but none of the callbacks are called and it doesn't appear in teams of course.

Is there any particular way to investigate what in particular is going wrong here? I initially suspected that the kubernetes deployment is not working somehow, but it seems to be a pure permission issue. This is weird though, because the bot has all call-related permissions and I triple-checked that all of them match that of the one already working:

• CallRecords.Read.All
• Calls.AccessMedia.All
• Calls.Initiate.All
• Calls.InitiateGroupCall.All
• Calls.JoinGroupCall.All
• Calls.JoinGroupCallAsGuest.All

All of those have been granted, both via link and via the "compound" confirmation link.

Still, the bot does not join. Some other issues were opened like #330 or #173, but none seem to offer a comprehensive solution.

Besides, we already used the same workflow to deploy succesfully once, so it seems highly suspicious that the second time around it doesn't work after a few simple renames.

Things like TLS also seem to work fine and I can send fake data to /api/calling which my AKS pods receive, so that isn't the issue either.

Start with the AKS deployment and compare it with the working deployment, then inspect the /communications/calls/CALLID/keepAlive logs and GraphAuthClient.cs around SendHttpRequestAsync. Check the callback path at /api/calling and the listed Graph call permissions. Done means identifying why the call is not found after creation and why the bot receives no callbacks.