Deploying blob storage with CMK and rbac-enabled keyVault fails due to race condition
- Dominant language
- TypeScript
- Stars
- 108
- Forks
- 44
- Avg merge
- 18h 53m
- Merged PRs (30d)
- 29
Description
**Bicep version**
Bicep CLI version 0.33.93 (7a77c7f2a5)
**Describe the bug**
Deploying blob storage with CMK and rbac-enabled keyVault fails due to a race condition. It seems that role assignments are applied inside storage managed identity only after a small delay. An explicit wait (using a deployment script) works around the issue (but is, of course, fragile).
**To Reproduce**
* Deploy an rbac-enabled keyVault containing a CMK
* give an MI permissions (rbac) to the keyVault
* create a key
* deploy blob storage using the key and with the MI as identity
The deployment will fail with the error message
`[{"code":"KeyVaultAuthenticationFailure","message":"The operation failed because of authentication issue on the keyvault.`
**Additional context**
* the issue does not occur if access policies are used instead of rbac on the keyVault
* the deployment is successful if the deployment is retried
This code does not work:
```bicep
@description('Specifies the name of the environment.')
param environment string
@description('Specifies the name of the deployment.')
@minLength(1)
@maxLength(10)
param customerAsset string
@description('Set soft delete retention in days for key vault')
param softDeleteRetentionInDays int
@description('We only use 2 letters as a unique suffix so that we adhere to the Naming Rules and Character Restrictions, see https://learn.microsoft.com/en-us/azure/azure-resource-manager/management/resource-name-rules')
var uniqueSuffix = substring(uniqueString(resourceGroup().id), 0, 2)
var region = 'gwc'
var storageAccountName = 'st${customerAsset}${environment}${region}${uniqueSuffix}'
var storageIdentityName = 'id-${customerAsset}-${environment}-${region}-storage${uniqueSuffix}'
var storageAccountKeyName = 'key-${customerAsset}-${environment}-${region}-storage${uniqueSuffix}'
var keyVaultName = 'kv-${customerAsset}-${environment}-${region}${uniqueSuffix}'
resource storageIdentity 'Microsoft.ManagedIdentity/userAssignedIdentities@2022-01-31-preview' = {
name: storageIdentityName
location: resourceGroup().location
tags: resourceGroup().tags
}
resource keyVault 'Microsoft.KeyVault/vaults@2023-07-01' = {
name: keyVaultName
location: resourceGroup().location
tags: resourceGroup().tags
properties: {
tenantId: subscription().tenantId
sku: {
name: 'standard'
family: 'A'
}
enablePurgeProtection: true
enableSoftDelete: true
softDeleteRetentionInDays: softDeleteRetentionInDays
enableRbacAuthorization: true
}
}
@description('This is the built-in Key Vault Crypto Service Encryption User role ')
resource keyVaultCryptoServiceEncryptionUserRole 'Microsoft.Authorization/roleDefinitions@2022-04-01' existing = {
name: 'e147488a-f6f5-4113-8e2d-b22465e65bf6'
}
resource keyVaultCryptoServiceEncryptionUserAssignment 'Microsoft.Authorization/roleAssignments@2022-04-01' = {
name: guid(keyVaultCryptoServiceEncryptionUserRole.id, storageIdentity.id, keyVault.id)
scope: keyVault
properties: {
description: 'Allow access to perform cryptographic operations using keys.'
roleDefinitionId: keyVaultCryptoServiceEncryptionUserRole.id
principalId: storageIdentity.properties.principalId
principalType: 'ServicePrincipal'
}
}
@description('This is the built-in Key Vault Reader role. See https://docs.microsoft.com/azure/role-based-access-control/built-in-roles/security#key-vault-reader')
resource keyVaultReaderRole 'Microsoft.Authorization/roleDefinitions@2022-04-01' existing = {
name: '21090545-7ca7-4776-b22c-e363652d74d2'
}
resource keyVaultReaderRoleAssignment 'Microsoft.Authorization/roleAssignments@2022-04-01' = {
name: guid(keyVaultReaderRole.id, storageIdentity.id, keyVault.id)
scope: keyVault
properties: {
description: 'Read metadata of key vaults and its certificates, keys, and secrets.'
roleDefinitionId: keyVaultReaderRole.id
principalId: storageIdentity.properties.principalId
principalType: 'ServicePrincipal'
}
}
resource storageAccountKey 'Microsoft.KeyVault/vaults/keys@2023-07-01' = {
parent: keyVault
name: storageAccountKeyName
tags: resourceGroup().tags
properties: {
kty: 'RSA'
keySize: 4096
rotationPolicy: {
lifetimeActions: [
{
trigger: {
timeAfterCreate: 'P1Y'
}
action: {
type: 'rotate'
}
}
{
trigger: {
timeBeforeExpiry: 'P30D'
}
action: {
type: 'notify'
}
}
]
}
attributes: {
enabled: true
}
}
}
resource storageAccount 'Microsoft.Storage/storageAccounts@2023-05-01' = {
name: storageAccountName
location: resourceGroup().location
tags: resourceGroup().tags
sku: {
name: 'Standard_LRS'
}
identity: {
type: 'UserAssigned'
userAssignedIdentities: { '${storageIdentity.id}': {} }
}
kind: 'StorageV2'
properties: {
accessTier: 'Hot'
supportsHttpsTrafficOnly: true
minimumTlsVersion: 'TLS1_2'
encryption: {
services: {
blob: {
enabled: true
keyType: 'Account'
}
file: {
enabled: true
keyType: 'Account'
}
queue: {
enabled: true
keyType: 'Account'
}
table: {
enabled: true
keyType: 'Account'
}
}
keySource: 'Microsoft.Keyvault'
identity: {
userAssignedIdentity: storageIdentity.id
}
keyvaultproperties: {
keyname: storageAccountKey.name
keyvaulturi: keyVault.properties.vaultUri
}
}
}
dependsOn: [
keyVaultReaderRoleAssignment
keyVaultCryptoServiceEncryptionUserAssignment
]
}
```
This is working code:
```bicep
@description('Specifies the name of the environment.')
param environment string
@description('Specifies the name of the deployment.')
@minLength(1)
@maxLength(10)
param customerAsset string
@description('Set soft delete retention in days for key vault')
param softDeleteRetentionInDays int
@description('We only use 2 letters as a unique suffix so that we adhere to the Naming Rules and Character Restrictions, see https://learn.microsoft.com/en-us/azure/azure-resource-manager/management/resource-name-rules')
var uniqueSuffix = substring(uniqueString(resourceGroup().id), 0, 2)
var region = 'gwc'
var storageAccountName = 'st${customerAsset}${environment}${region}${uniqueSuffix}'
var storageIdentityName = 'id-${customerAsset}-${environment}-${region}-storage${uniqueSuffix}'
var storageAccountKeyName = 'key-${customerAsset}-${environment}-${region}-storage${uniqueSuffix}'
var keyVaultName = 'kv-${customerAsset}-${environment}-${region}${uniqueSuffix}'
resource storageIdentity 'Microsoft.ManagedIdentity/userAssignedIdentities@2022-01-31-preview' = {
name: storageIdentityName
location: resourceGroup().location
tags: resourceGroup().tags
}
resource keyVault 'Microsoft.KeyVault/vaults@2023-07-01' = {
name: keyVaultName
location: resourceGroup().location
tags: resourceGroup().tags
properties: {
tenantId: subscription().tenantId
sku: {
name: 'standard'
family: 'A'
}
/* networkAcls: networkAcls */
enablePurgeProtection: true
enableSoftDelete: true
softDeleteRetentionInDays: softDeleteRetentionInDays
enableRbacAuthorization: true
}
}
@description('This is the built-in Key Vault Crypto Service Encryption User role ')
resource keyVaultCryptoServiceEncryptionUserRole 'Microsoft.Authorization/roleDefinitions@2022-04-01' existing = {
name: 'e147488a-f6f5-4113-8e2d-b22465e65bf6'
}
resource keyVaultCryptoServiceEncryptionUserAssignment 'Microsoft.Authorization/roleAssignments@2022-04-01' = {
name: guid(keyVaultCryptoServiceEncryptionUserRole.id, storageIdentity.id, keyVault.id)
scope: keyVault
properties: {
description: 'Allow access to perform cryptographic operations using keys.'
roleDefinitionId: keyVaultCryptoServiceEncryptionUserRole.id
principalId: storageIdentity.properties.principalId
principalType: 'ServicePrincipal'
}
}
@description('This is the built-in Key Vault Reader role. See https://docs.microsoft.com/azure/role-based-access-control/built-in-roles/security#key-vault-reader')
resource keyVaultReaderRole 'Microsoft.Authorization/roleDefinitions@2022-04-01' existing = {
name: '21090545-7ca7-4776-b22c-e363652d74d2'
}
resource keyVaultReaderRoleAssignment 'Microsoft.Authorization/roleAssignments@2022-04-01' = {
name: guid(keyVaultReaderRole.id, storageIdentity.id, keyVault.id)
scope: keyVault
properties: {
description: 'Read metadata of key vaults and its certificates, keys, and secrets.'
roleDefinitionId: keyVaultReaderRole.id
principalId: storageIdentity.properties.principalId
principalType: 'ServicePrincipal'
}
}
resource storageAccountKey 'Microsoft.KeyVault/vaults/keys@2023-07-01' = {
parent: keyVault
name: storageAccountKeyName
tags: resourceGroup().tags
properties: {
kty: 'RSA'
keySize: 4096
rotationPolicy: {
lifetimeActions: [
{
trigger: {
timeAfterCreate: 'P1Y'
}
action: {
type: 'rotate'
}
}
{
trigger: {
timeBeforeExpiry: 'P30D'
}
action: {
type: 'notify'
}
}
]
}
attributes: {
enabled: true
}
}
}
resource deploymentScript 'Microsoft.Resources/deploymentScripts@2023-08-01' = {
name: 'inlineCLI'
location: resourceGroup().location
kind: 'AzureCLI'
properties: {
azCliVersion: '2.52.0'
scriptContent: '''
sleep 10s
'''
retentionInterval: 'PT1H'
}
dependsOn: [
keyVaultReaderRoleAssignment
keyVaultCryptoServiceEncryptionUserAssignment
]
}
resource storageAccount 'Microsoft.Storage/storageAccounts@2023-05-01' = {
name: storageAccountName
location: resourceGroup().location
tags: resourceGroup().tags
sku: {
name: 'Standard_LRS'
}
identity: {
type: 'UserAssigned'
userAssignedIdentities: { '${storageIdentity.id}': {} }
}
kind: 'StorageV2'
properties: {
accessTier: 'Hot'
supportsHttpsTrafficOnly: true
minimumTlsVersion: 'TLS1_2'
encryption: {
services: {
blob: {
enabled: true
keyType: 'Account'
}
file: {
enabled: true
keyType: 'Account'
}
queue: {
enabled: true
keyType: 'Account'
}
table: {
enabled: true
keyType: 'Account'
}
}
keySource: 'Microsoft.Keyvault'
identity: {
userAssignedIdentity: storageIdentity.id
}
keyvaultproperties: {
keyname: storageAccountKey.name
keyvaulturi: keyVault.properties.vaultUri
}
}
}
dependsOn: [
deploymentScript
]
}
```
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by reproducing the deployment from the failing Bicep example with an RBAC-enabled Key Vault, customer-managed key, and user-assigned identity, then compare it with the deployment-script workaround. The issue is done when the storage deployment succeeds without the explicit wait; no repository file or test entry point is identified in the report.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- azure
- Domain
- cloud, infrastructure
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100