Unable to establish connection to KeyVault due to race condition when connecting via VNet (private IP isn't used when it should be to make connection)

I'm running into a strange, inconsistent issue where autoscaling seems to be corrupting the function app's access to Key Vault.

In some cases where the App Service Plan is automatically scaled out, none of my key vault references in my configurations are able to resolve. When I look at the reason, it says "Other Reasons".

Only fix I've found was having to delete the access policy for the function app in Key Vault and re-add it again. Once I do this then upon checking on the function app, the references are resolved.

This is obviously not ideal as it makes the autoscaling feature of the App Service plan useless as I sometimes need to manually intervene.

I first came across the issue when I UpScaled the App Service Plan and noticed that the Key Vault references were not resolving. I initially thought it had to do with the Private Endpoint and VNet integration that was set up for the function app.

Also there are quite a few function apps in the App Service Plan, only about 3 of them would get the Key Vault reference issue. The others would still be working after the autoscale.
Also the issue only seems to occur once out of every 6 or so autoscale instances. So its not consistent.

Any help on this issue would be appreciated.