Azure / Azure/AKS-Construction

CI/CD Deployment testing failure

Open
#648 5 comments 0 reactions 0 assignees View on GitHub
bug needs-discussion
Dominant language
JavaScript
Stars
377
Forks
170
PR merge metrics
No merged PRs in 30d

Description

**Describe the bug**
During full deployment tests in the CI/CD pipeline, we get an error because of the state of the environment we're deploying to.

**To Reproduce**
1. Tag a PR with either;
[test-deploy-byoconfig](https://github.com/Azure/AKS-Construction/labels/test-deploy-byoconfig)
[test-deploy-privateconfig](https://github.com/Azure/AKS-Construction/labels/test-deploy-privateconfig)
2. Wait for checks to run and fail.

**Expected behavior**
Environment considerations are properly reset so deployment tests can run.

**Additional context**
```output
ERROR: {"status":"Failed","error":{"code":"DeploymentFailed","message":"At least one resource deployment operation failed.
Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.",
"details":[{"code":"BadRequest","message":"{\r\n \"error\": {\r\n
\"code\": \"RoleAssignmentUpdateNotPermitted\",\r\n
\"message\": \"Tenant ID, application ID, principal ID, and scope are not allowed to be updated.\"\r\n }\r\n}"}]}}
```

The role assignment thats having the problem is the RG Reader role for AppGw;

```bicep
// AGIC's identity requires "Reader" permission over Application Gateway's resource group.
var reader = subscriptionResourceId('Microsoft.Authorization/roleDefinitions', 'acdd72a7-3385-48ef-bd42-f606fba81ae7')
resource appGwAGICRGReader 'Microsoft.Authorization/roleAssignments@2022-04-01' = if (ingressApplicationGateway && deployAppGw) {
scope: resourceGroup()
name: guid(aks.id, 'Agic', reader)
properties: {
roleDefinitionId: reader
principalType: 'ServicePrincipal'
principalId: aks.properties.addonProfiles.ingressApplicationGateway.identity.objectId
}
}
```

The reason it's having a problem is because the name isn't unique. It's using a static 'Agic' string instead of an identifier to the identity such as principalId. This is because the identity is not known before main.bicep is launched, therefore it cannot form part of the name.

I see 3 options for resolution;
1. Refactor the AppGw out of main.bicep, which will allow the role assignment name guid to be based on the already existing AKS AppGW ObjectId.
2. Debug why the environment cleanup is not removing this role assignment (it should). Implement fix.
3. Speak to the AKS team about addOn support for existing managed identities

Contributor guide

Open the contributing guide

Research direction

Start in main.bicep at appGwAGICRGReader and reproduce the failure by tagging a PR with test-deploy-byoconfig or test-deploy-privateconfig. Inspect deployment cleanup and the role-assignment naming behavior; the issue lists three possible resolution paths, so confirm the chosen path with maintainers before changing anything. Done means deployment tests pass without the RoleAssignmentUpdateNotPermitted error.

Written by the indexing model from the issue text.

Assessment

Tech stack
azure, kubernetes
Domain
ci-cd, cloud, infrastructure
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.