packer/image build as part of Azure GenerateResourceandImages.ps fails on private IP

I have had the process to generate a windows2022 image from the GenerateResourceandImage process for a few years. (part of the git-imagerunners)
The VNET we used in Azure was open and connections to packer to pull down the bits worked fine.

I have a new resource group in Azure that has a defaulting VNET. I have relaxed the service endpoints so there is no keyvault firewall.
I created a new VNET/subnet to look like the one we were using.
I setup the packer.json to specify, the VNET name, VNET subnet name, and build image resource group.

When I run the release to execute the process, I get the following error:

Getting the certificate's URL ...
2024-02-27T20:39:54.8581459Z ==> azure-arm.image: -> Key Vault Name : 'pkrkvztifzt2p81'
2024-02-27T20:39:54.8583086Z ==> azure-arm.image: -> Key Vault Secret Name : 'packerKeyVaultSecret'
2024-02-27T20:39:54.9360793Z ==> azure-arm.image: ERROR: -> Forbidden : Client address is not authorized and caller is not a trusted service.
2024-02-27T20:39:54.9361545Z ==> azure-arm.image: Client address: x.x.x.x from unknown subnet
2024-02-27T20:39:54.9364202Z ==> azure-arm.image: Caller: appid=***;oid=29e1fbed-ab6d-4741-8d3f-48590f5943c0;iss=https://sts.windows.net/***/
2024-02-27T20:39:54.9364737Z ==> azure-arm.image: Vault: pkrkvztifzt2p81;location=eastus2

where x.x.x.x=private IP address of our on prem(in Azure) build agent.

Do we know how to resolve this, and why its looking at the private IP vs public when the VNET is open?

The issue names the GenerateResourceandImages process and packer.json; start by reviewing those inputs and reproducing the release against the private-IP VNET. Trace the Azure Key Vault request that returns Forbidden and verify which client address and subnet authorization it uses. Done means the Windows 2022 image build succeeds on the private network, or the required Azure configuration change is documented.

Written by the indexing model from the issue text.