GoogleCloudPlatform / GoogleCloudPlatform/pubsec-declarative-toolkit
Regression: 1 of 3 namespaces in hub-env service-account.yaml is kpt-set to config-control instead of hardcoded networking
- Dominant language
- Shell
- Stars
- 36
- Forks
- 26
- PR merge metrics
- No merged PRs in 30d
Description
part of #446
Following
https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit/blob/main/solutions/project/hub-env/fortigate/service-account.yaml#L36C1-L37C63
```
name: fortigatesdn-sa-fortigatesdnviewer-role-permissions
-namespace: config-control # kpt-set: ${management-namespace}
+namespace: networking
```
must match
https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit/blob/main/solutions/project/hub-env/fortigate/management-vm/service-account.yaml#L37
```
name: hub-admin-serviceaccountuser-permissions
namespace: networking
```
SAs for hub are in the networking namespace instead of the config-control one - one of 3 resources is left in config-control - fix is to remove the kpt-set and hardcode to networking
Notes:
further, note that my env is clean - like clients so some of these issues only occur once - like roles on the GKE service accounts.hub project comes up like my 1st envthere was one issue blocking SA permissions - 1 of 4 namespaces for both service-account.yaml had a kpt-set and was hardcoded to config-control - while the other 3 were in networking namespacethis was due to a missing 4th setters.yamlmanagement-namespace: config-control
had to remove the kpt-set so one of the 4 does not stay in config-control
```
metadata: # kpt-merge: networking/hub-managementvm-sa
name: hub-managementvm-sa
namespace: networking
metadata: # kpt-merge: networking/hub-fortigatesdn-sa
name: hub-fortigatesdn-sa
namespace: networking
found the issue
name: fortigatesdn-sa-fortigatesdnviewer-role-permissions
namespace: config-control # kpt-set: ${management-namespace}
should be networking namespace
this would fix the fact that the IAMPolicyMember is in the config-control namespace but it's service account is in the networking namespace
to summarize
hub-env/fortigate/service-account.yaml
- sa in networking
- policy in config-control
hub-env/fortigate/management-vm/service-account.yaml
- sa in networking
- policy in networking
fix is to move the
fortigatesdn-sa-fortigatesdnviewer-role-permissions
to the network namespace
metadata: # kpt-merge: config-control/fortigatesdn-sa-fortigatesdnviewer-role-permissions
name: fortigatesdn-sa-fortigatesdnviewer-role-permissions
namespace: config-control # kpt-set: ${management-namespace}
to
metadata: # kpt-merge: networking/fortigatesdn-sa-fortigatesdnviewer-role-permissions
name: fortigatesdn-sa-fortigatesdnviewer-role-permissions
namespace: config-control # kpt-set: ${management-namespace}
to match
metadata: # kpt-merge: networking/hub-admin-serviceaccountuser-permissions
name: hub-admin-serviceaccountuser-permissions
namespace: networking
and add
management-namespace: networking
to setters.yaml
```
add to setters
```
management-namespace: networking
fixes
namespace: networking # kpt-set: ${management-namespace}
```
```
but breaks the VM's - we will hardcode instead like the management-vm/service-account.yaml
michael@cloudshell:~/kcc-oi/kpt (kcc-oi-3552)$ kpt live apply hub-env
installing inventory ResourceGroup CRD.
error: 5 errors:
- invalid object: "networking_hub-fgt-primary-log-disk_compute.cnrm.cloud.google.com_ComputeDisk": invalid "config.kubernetes.io/depends-on" annotation: external dependency: compute.cnrm.cloud.google.com/namespaces/networking/ComputeDisk/hub-fgt-primary-log-disk -> iam.cnrm.cloud.google.com/namespaces/config-control/IAMPolicyMember/networking-sa-computeinstanceadmin-permissions
- invalid object: "networking_hub-fgt-secondary-log-disk_compute.cnrm.cloud.google.com_ComputeDisk": invalid "config.kubernetes.io/depends-on" annotation: external dependency: compute.cnrm.cloud.google.com/namespaces/networking/ComputeDisk/hub-fgt-secondary-log-disk -> iam.cnrm.cloud.google.com/namespaces/config-control/IAMPolicyMember/networking-sa-computeinstanceadmin-permissions
- invalid object: "networking_hub-http-8008-httphc_compute.cnrm.cloud.google.com_ComputeHTTPHealthCheck": invalid "config.kubernetes.io/depends-on" annotation: external dependency: compute.cnrm.cloud.google.com/namespaces/networking/ComputeHTTPHealthCheck/hub-http-8008-httphc -> iam.cnrm.cloud.google.com/namespaces/config-control/IAMPolicyMember/networking-sa-computeinstanceadmin-permissions
- invalid object: "networking_hub-http-8008-hc_compute.cnrm.cloud.google.com_ComputeHealthCheck": invalid "config.kubernetes.io/depends-on" annotation: external dependency: compute.cnrm.cloud.google.com/namespaces/networking/ComputeHealthCheck/hub-http-8008-hc -> iam.cnrm.cloud.google.com/namespaces/config-control/IAMPolicyMember/networking-sa-computeinstanceadmin-permissions
- invalid object: "networking_hub-mgmt-data-disk_compute.cnrm.cloud.google.com_ComputeDisk": invalid "config.kubernetes.io/depends-on" annotation: external dependency: compute.cnrm.cloud.google.com/namespaces/networking/ComputeDisk/hub-mgmt-data-disk -> iam.cnrm.cloud.google.com/namespaces/config-control/IAMPolicyMember/networking-sa-computeinstanceadmin-permissions
and also hardcoded ones in project-iam.yaml
namespace: config-control # kpt-set: ${management-namespace}
```
fixed
```
michael@cloudshell:~/kcc-oi/kpt (kcc-oi-3552)$ kpt live apply hub-env
installing inventory ResourceGroup CRD.
error: 5 errors:
- invalid object: "networking_hub-fgt-primary-log-disk_compute.cnrm.cloud.google.com_ComputeDisk": invalid "config.kubernetes.io/depends-on" annotation: external dependency: compute.cnrm.cloud.google.com/namespaces/networking/ComputeDisk/hub-fgt-primary-log-disk -> iam.cnrm.cloud.google.com/namespaces/config-control/IAMPolicyMember/networking-sa-computeinstanceadmin-permissions
- invalid object: "networking_hub-fgt-secondary-log-disk_compute.cnrm.cloud.google.com_ComputeDisk": invalid "config.kubernetes.io/depends-on" annotation: external dependency: compute.cnrm.cloud.google.com/namespaces/networking/ComputeDisk/hub-fgt-secondary-log-disk -> iam.cnrm.cloud.google.com/namespaces/config-control/IAMPolicyMember/networking-sa-computeinstanceadmin-permissions
- invalid object: "networking_hub-http-8008-httphc_compute.cnrm.cloud.google.com_ComputeHTTPHealthCheck": invalid "config.kubernetes.io/depends-on" annotation: external dependency: compute.cnrm.cloud.google.com/namespaces/networking/ComputeHTTPHealthCheck/hub-http-8008-httphc -> iam.cnrm.cloud.google.com/namespaces/config-control/IAMPolicyMember/networking-sa-computeinstanceadmin-permissions
- invalid object: "networking_hub-http-8008-hc_compute.cnrm.cloud.google.com_ComputeHealthCheck": invalid "config.kubernetes.io/depends-on" annotation: external dependency: compute.cnrm.cloud.google.com/namespaces/networking/ComputeHealthCheck/hub-http-8008-hc -> iam.cnrm.cloud.google.com/namespaces/config-control/IAMPolicyMember/networking-sa-computeinstanceadmin-permissions
- invalid object: "networking_hub-mgmt-data-disk_compute.cnrm.cloud.google.com_ComputeDisk": invalid "config.kubernetes.io/depends-on" annotation: external dependency: compute.cnrm.cloud.google.com/namespaces/networking/ComputeDisk/hub-mgmt-data-disk -> iam.cnrm.cloud.google.com/namespaces/config-control/IAMPolicyMember/networking-sa-computeinstanceadmin-permissions
michael@cloudshell:~/kcc-oi/kpt (kcc-oi-3552)$ kpt fn render hub-env
Package "hub-env":
[RUNNING] "gcr.io/kpt-fn/apply-setters:v0.2"
[PASS] "gcr.io/kpt-fn/apply-setters:v0.2" in 500ms
Results:
[info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "xxdmu-admin1-hub-oi9"
[info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "xxdmu-admin1-hub-oi9"
[info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "xxdmu-admin1-hub-oi9"
[info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "xxdmu-admin1-hub-oi9"
...(113 line(s) truncated, use '--truncate-output=false' to disable)
[RUNNING] "gcr.io/kpt-fn/search-replace:v0.2.0"
[PASS] "gcr.io/kpt-fn/search-replace:v0.2.0" in 400ms
Results:
[info]: no matches
Successfully executed 2 function(s) in 1 package(s).
michael@cloudshell:~/kcc-oi/kpt (kcc-oi-3552)$ kpt live apply hub-env
installing inventory ResourceGroup CRD.
inventory update started
inventory update finished
apply phase started
computeinstance.compute.cnrm.cloud.google.com/hub-fgt-primary-instance apply successful
computeinstance.compute.cnrm.cloud.google.com/hub-fgt-secondary-instance apply successful
iampolicymember.iam.cnrm.cloud.google.com/fortigatesdn-sa-fortigatesdnviewer-role-permissions apply successful
iampolicymember.iam.cnrm.cloud.google.com/hub-admin-serviceaccountuser-permissions apply successful
iamserviceaccount.iam.cnrm.cloud.google.com/hub-fortigatesdn-sa apply successful
iamserviceaccount.iam.cnrm.cloud.google.com/hub-managementvm-sa apply successful
project.resourcemanager.cnrm.cloud.google.com/xxdmu-admin1-hub-oi9 apply successful
apply phase finished
reconcile phase started
computeinstance.compute.cnrm.cloud.google.com/hub-fgt-primary-instance reconcile pending
computeinstance.compute.cnrm.cloud.google.com/hub-fgt-secondary-instance reconcile pending
iampolicymember.iam.cnrm.cloud.google.com/fortigatesdn-sa-fortigatesdnviewer-role-permissions reconcile pending
iampolicymember.iam.cnrm.cloud.google.com/hub-admin-serviceaccountuser-permissions reconcile pending
iamserviceaccount.iam.cnrm.cloud.google.com/hub-fortigatesdn-sa reconcile pending
iamserviceaccount.iam.cnrm.cloud.google.com/hub-managementvm-sa reconcile pending
project.resourcemanager.cnrm.cloud.google.com/xxdmu-admin1-hub-oi9 reconcile successful
destroy
recreate
moved over
michael@cloudshell:~/kcc-oi/kpt (kcc-oi-3552)$ kubectl get gcp -n networking
NAME AGE READY STATUS STATUS AGE
computeinstance.compute.cnrm.cloud.google.com/hub-fgt-primary-instance 79s False DependencyNotFound 79s
computeinstance.compute.cnrm.cloud.google.com/hub-fgt-secondary-instance 79s False DependencyNotFound 79s
NAME AGE READY STATUS STATUS AGE
iampolicymember.iam.cnrm.cloud.google.com/fortigatesdn-sa-fortigatesdnviewer-role-permissions 82s False UpdateFailed 81s
iampolicymember.iam.cnrm.cloud.google.com/hub-admin-serviceaccountuser-permissions 81s False DependencyNotReady 81s
NAME AGE READY STATUS STATUS AGE
iamserviceaccount.iam.cnrm.cloud.google.com/hub-fortigatesdn-sa 81s False UpdateFailed 81s
iamserviceaccount.iam.cnrm.cloud.google.com/hub-managementvm-sa 81s False UpdateFailed 81s
further
Warning DependencyNotFound 2m19s computeinstance-controller reference ComputeDisk networking/hub-fgt-primary-log-disk is not found
michael@cloudshell:~/kcc-oi/kpt (kcc-oi-3552)$ kubectl describe computeinstance.compute.cnrm.cloud.google.com/hub-fgt-primary-instance -n networking
Warning UpdateFailed 75s (x7 over 3m22s) iampolicymember-controller Update call failed: error fetching live state for resource: error reading underlying resource: summary: Error when reading or editing Resource "organization \"459065442144\"" with IAM Member: Role "organizations/459065442144/roles/FortigateSdnViewer" Member "serviceAccount:fortigatesdn-sa@xxdmu-admin1-hub-oi10.iam.gserviceaccount.com": Error retrieving IAM policy for organization "459065442144": googleapi: Error 403: The caller does not have permission, forbidden
```
```
need to add Organization Role Admin (and update the script to retrieve the account set the role before rendering
```
```
Status:
Conditions:
Last Transition Time: 2023-10-23T23:11:05Z
Message: Update call failed: error applying desired state: summary: Error creating service account: googleapi: Error 403: Permission 'iam.serviceAccounts.create' denied on resource (or it may not exist).
```
also need to add
"Create Service Accounts" role
destroy/recreate because of immutable SAs
```
apiVersion: v1
kind: ConfigMap
metadata:
name: setters
annotations:
config.kubernetes.io/local-config: "true"
data:
org-id: "459065442144"
project-billing-id: "014479-806359-2F5F85"
project-parent-folder: services-infrastructure
hub-project-id: xxdmu-admin1-hub-oi11
management-project-id: xxdmu-admin1-hub-oi11
# must be config-control
management-namespace: config-control
hub-admin: group:sas@obrien.industries
project-allowed-restrict-vpc-peering: |
- under:organizations/459065442144
project-allowed-vm-external-ip-access: |
- "projects/xxdmu-admin1-hub-oi11/zones/northamerica-northeast1-a/instances/fgt-primary-instance"
- "projects/xxdmu-admin1-hub-oi11/zones/northamerica-northeast1-b/instances/fgt-secondary-instance"
project-allowed-vm-can-ip-forward: |
- "projects/xxdmu-admin1-hub-oi11/zones/northamerica-northeast1-a/instances/fgt-primary-instance"
- "projects/xxdmu-admin1-hub-oi11/zones/northamerica-northeast1-b/instances/fgt-secondary-instance"
fgt-primary-image: projects/fortigcp-project-001/global/images/fortinet-fgtondemand-724-20230201-001-w-license
fgt-primary-license: |
LICENSE
fgt-secondary-image: projects/fortigcp-project-001/global/images/fortinet-fgtondemand-724-20230201-001-w-license
fgt-secondary-license: |
LICENSE
```
Contributor guide
Assessment
This issue has not been assessed yet.