Azure / Azure/bicep-types-az

Deploying blob storage with CMK and rbac-enabled keyVault fails due to race condition

Open
#2,452 1 comment 0 reactions 0 assignees View on GitHub
Needs: Triage :mag:
Dominant language
TypeScript
Stars
108
Forks
44
Avg merge
18h 53m
Merged PRs (30d)
29

Description

**Bicep version**
Bicep CLI version 0.33.93 (7a77c7f2a5)

**Describe the bug**

Deploying blob storage with CMK and rbac-enabled keyVault fails due to a race condition. It seems that role assignments are applied inside storage managed identity only after a small delay. An explicit wait (using a deployment script) works around the issue (but is, of course, fragile).

**To Reproduce**

* Deploy an rbac-enabled keyVault containing a CMK
* give an MI permissions (rbac) to the keyVault
* create a key
* deploy blob storage using the key and with the MI as identity

The deployment will fail with the error message

`[{"code":"KeyVaultAuthenticationFailure","message":"The operation failed because of authentication issue on the keyvault.`

**Additional context**

* the issue does not occur if access policies are used instead of rbac on the keyVault
* the deployment is successful if the deployment is retried

This code does not work:
```bicep
@description('Specifies the name of the environment.')
param environment string

@description('Specifies the name of the deployment.')
@minLength(1)
@maxLength(10)
param customerAsset string

@description('Set soft delete retention in days for key vault')
param softDeleteRetentionInDays int

@description('We only use 2 letters as a unique suffix so that we adhere to the Naming Rules and Character Restrictions, see https://learn.microsoft.com/en-us/azure/azure-resource-manager/management/resource-name-rules')
var uniqueSuffix = substring(uniqueString(resourceGroup().id), 0, 2)
var region = 'gwc'
var storageAccountName = 'st${customerAsset}${environment}${region}${uniqueSuffix}'
var storageIdentityName = 'id-${customerAsset}-${environment}-${region}-storage${uniqueSuffix}'
var storageAccountKeyName = 'key-${customerAsset}-${environment}-${region}-storage${uniqueSuffix}'
var keyVaultName = 'kv-${customerAsset}-${environment}-${region}${uniqueSuffix}'

resource storageIdentity 'Microsoft.ManagedIdentity/userAssignedIdentities@2022-01-31-preview' = {
name: storageIdentityName
location: resourceGroup().location
tags: resourceGroup().tags
}

resource keyVault 'Microsoft.KeyVault/vaults@2023-07-01' = {
name: keyVaultName
location: resourceGroup().location
tags: resourceGroup().tags
properties: {
tenantId: subscription().tenantId
sku: {
name: 'standard'
family: 'A'
}
enablePurgeProtection: true
enableSoftDelete: true
softDeleteRetentionInDays: softDeleteRetentionInDays
enableRbacAuthorization: true
}
}

@description('This is the built-in Key Vault Crypto Service Encryption User role ')
resource keyVaultCryptoServiceEncryptionUserRole 'Microsoft.Authorization/roleDefinitions@2022-04-01' existing = {
name: 'e147488a-f6f5-4113-8e2d-b22465e65bf6'
}

resource keyVaultCryptoServiceEncryptionUserAssignment 'Microsoft.Authorization/roleAssignments@2022-04-01' = {
name: guid(keyVaultCryptoServiceEncryptionUserRole.id, storageIdentity.id, keyVault.id)
scope: keyVault
properties: {
description: 'Allow access to perform cryptographic operations using keys.'
roleDefinitionId: keyVaultCryptoServiceEncryptionUserRole.id
principalId: storageIdentity.properties.principalId
principalType: 'ServicePrincipal'
}
}

@description('This is the built-in Key Vault Reader role. See https://docs.microsoft.com/azure/role-based-access-control/built-in-roles/security#key-vault-reader')
resource keyVaultReaderRole 'Microsoft.Authorization/roleDefinitions@2022-04-01' existing = {
name: '21090545-7ca7-4776-b22c-e363652d74d2'
}

resource keyVaultReaderRoleAssignment 'Microsoft.Authorization/roleAssignments@2022-04-01' = {
name: guid(keyVaultReaderRole.id, storageIdentity.id, keyVault.id)
scope: keyVault
properties: {
description: 'Read metadata of key vaults and its certificates, keys, and secrets.'
roleDefinitionId: keyVaultReaderRole.id
principalId: storageIdentity.properties.principalId
principalType: 'ServicePrincipal'
}
}

resource storageAccountKey 'Microsoft.KeyVault/vaults/keys@2023-07-01' = {
parent: keyVault
name: storageAccountKeyName
tags: resourceGroup().tags
properties: {
kty: 'RSA'
keySize: 4096
rotationPolicy: {
lifetimeActions: [
{
trigger: {
timeAfterCreate: 'P1Y'
}
action: {
type: 'rotate'
}
}
{
trigger: {
timeBeforeExpiry: 'P30D'
}
action: {
type: 'notify'
}
}
]
}
attributes: {
enabled: true
}
}
}

resource storageAccount 'Microsoft.Storage/storageAccounts@2023-05-01' = {
name: storageAccountName
location: resourceGroup().location
tags: resourceGroup().tags
sku: {
name: 'Standard_LRS'
}
identity: {
type: 'UserAssigned'
userAssignedIdentities: { '${storageIdentity.id}': {} }
}
kind: 'StorageV2'
properties: {
accessTier: 'Hot'
supportsHttpsTrafficOnly: true
minimumTlsVersion: 'TLS1_2'
encryption: {
services: {
blob: {
enabled: true
keyType: 'Account'
}
file: {
enabled: true
keyType: 'Account'
}
queue: {
enabled: true
keyType: 'Account'
}
table: {
enabled: true
keyType: 'Account'
}
}
keySource: 'Microsoft.Keyvault'
identity: {
userAssignedIdentity: storageIdentity.id
}
keyvaultproperties: {
keyname: storageAccountKey.name
keyvaulturi: keyVault.properties.vaultUri
}
}
}
dependsOn: [
keyVaultReaderRoleAssignment
keyVaultCryptoServiceEncryptionUserAssignment
]
}

```

This is working code:

```bicep
@description('Specifies the name of the environment.')
param environment string

@description('Specifies the name of the deployment.')
@minLength(1)
@maxLength(10)
param customerAsset string

@description('Set soft delete retention in days for key vault')
param softDeleteRetentionInDays int

@description('We only use 2 letters as a unique suffix so that we adhere to the Naming Rules and Character Restrictions, see https://learn.microsoft.com/en-us/azure/azure-resource-manager/management/resource-name-rules')
var uniqueSuffix = substring(uniqueString(resourceGroup().id), 0, 2)
var region = 'gwc'
var storageAccountName = 'st${customerAsset}${environment}${region}${uniqueSuffix}'
var storageIdentityName = 'id-${customerAsset}-${environment}-${region}-storage${uniqueSuffix}'
var storageAccountKeyName = 'key-${customerAsset}-${environment}-${region}-storage${uniqueSuffix}'
var keyVaultName = 'kv-${customerAsset}-${environment}-${region}${uniqueSuffix}'

resource storageIdentity 'Microsoft.ManagedIdentity/userAssignedIdentities@2022-01-31-preview' = {
name: storageIdentityName
location: resourceGroup().location
tags: resourceGroup().tags
}

resource keyVault 'Microsoft.KeyVault/vaults@2023-07-01' = {
name: keyVaultName
location: resourceGroup().location
tags: resourceGroup().tags
properties: {
tenantId: subscription().tenantId
sku: {
name: 'standard'
family: 'A'
}
/* networkAcls: networkAcls */
enablePurgeProtection: true
enableSoftDelete: true
softDeleteRetentionInDays: softDeleteRetentionInDays
enableRbacAuthorization: true
}
}

@description('This is the built-in Key Vault Crypto Service Encryption User role ')
resource keyVaultCryptoServiceEncryptionUserRole 'Microsoft.Authorization/roleDefinitions@2022-04-01' existing = {
name: 'e147488a-f6f5-4113-8e2d-b22465e65bf6'
}

resource keyVaultCryptoServiceEncryptionUserAssignment 'Microsoft.Authorization/roleAssignments@2022-04-01' = {
name: guid(keyVaultCryptoServiceEncryptionUserRole.id, storageIdentity.id, keyVault.id)
scope: keyVault
properties: {
description: 'Allow access to perform cryptographic operations using keys.'
roleDefinitionId: keyVaultCryptoServiceEncryptionUserRole.id
principalId: storageIdentity.properties.principalId
principalType: 'ServicePrincipal'
}
}

@description('This is the built-in Key Vault Reader role. See https://docs.microsoft.com/azure/role-based-access-control/built-in-roles/security#key-vault-reader')
resource keyVaultReaderRole 'Microsoft.Authorization/roleDefinitions@2022-04-01' existing = {
name: '21090545-7ca7-4776-b22c-e363652d74d2'
}

resource keyVaultReaderRoleAssignment 'Microsoft.Authorization/roleAssignments@2022-04-01' = {
name: guid(keyVaultReaderRole.id, storageIdentity.id, keyVault.id)
scope: keyVault
properties: {
description: 'Read metadata of key vaults and its certificates, keys, and secrets.'
roleDefinitionId: keyVaultReaderRole.id
principalId: storageIdentity.properties.principalId
principalType: 'ServicePrincipal'
}
}

resource storageAccountKey 'Microsoft.KeyVault/vaults/keys@2023-07-01' = {
parent: keyVault
name: storageAccountKeyName
tags: resourceGroup().tags
properties: {
kty: 'RSA'
keySize: 4096
rotationPolicy: {
lifetimeActions: [
{
trigger: {
timeAfterCreate: 'P1Y'
}
action: {
type: 'rotate'
}
}
{
trigger: {
timeBeforeExpiry: 'P30D'
}
action: {
type: 'notify'
}
}
]
}
attributes: {
enabled: true
}
}
}

resource deploymentScript 'Microsoft.Resources/deploymentScripts@2023-08-01' = {
name: 'inlineCLI'
location: resourceGroup().location
kind: 'AzureCLI'
properties: {
azCliVersion: '2.52.0'
scriptContent: '''
sleep 10s
'''
retentionInterval: 'PT1H'
}
dependsOn: [
keyVaultReaderRoleAssignment
keyVaultCryptoServiceEncryptionUserAssignment
]
}

resource storageAccount 'Microsoft.Storage/storageAccounts@2023-05-01' = {
name: storageAccountName
location: resourceGroup().location
tags: resourceGroup().tags
sku: {
name: 'Standard_LRS'
}
identity: {
type: 'UserAssigned'
userAssignedIdentities: { '${storageIdentity.id}': {} }
}
kind: 'StorageV2'
properties: {
accessTier: 'Hot'
supportsHttpsTrafficOnly: true
minimumTlsVersion: 'TLS1_2'
encryption: {
services: {
blob: {
enabled: true
keyType: 'Account'
}
file: {
enabled: true
keyType: 'Account'
}
queue: {
enabled: true
keyType: 'Account'
}
table: {
enabled: true
keyType: 'Account'
}
}
keySource: 'Microsoft.Keyvault'
identity: {
userAssignedIdentity: storageIdentity.id
}
keyvaultproperties: {
keyname: storageAccountKey.name
keyvaulturi: keyVault.properties.vaultUri
}
}
}
dependsOn: [
deploymentScript
]
}
```

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reproducing the deployment from the failing Bicep example with an RBAC-enabled Key Vault, customer-managed key, and user-assigned identity, then compare it with the deployment-script workaround. The issue is done when the storage deployment succeeds without the explicit wait; no repository file or test entry point is identified in the report.

Written by the indexing model from the issue text.

Assessment

Tech stack
azure
Domain
cloud, infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.