GoogleCloudPlatform / GoogleCloudPlatform/cluster-toolkit
GKE monitoring component ordering causes persistent Terraform drift
- Dominant language
- Go
- Stars
- 361
- Forks
- 322
- Avg merge
- 2d 21h
- Merged PRs (30d)
- 115
Description
### Describe the bug
The shared `modules/scheduler/gke-cluster` module produces a persistent Terraform diff when GKE returns `monitoring_config.enable_components` in a different order from the configured list. Reapplying succeeds but does not remove the diff, causing unnecessary cluster updates and failing integration tests that require an empty plan.
### Steps to reproduce
1. Deploy a regional GKE Standard cluster using the module's default `monitoring_components` and `enable_dcgm_monitoring` settings.
2. Reapply the deployment without changing its configuration.
3. Run `terraform plan -detailed-exitcode`.
### Expected behavior
An unchanged set of monitoring components should produce an empty plan (exit 0), while actual additions and removals remain managed by Terraform.
### Actual behavior
The plan exits 2 and proposes an in-place cluster update. The component membership is unchanged, only ordering differs. In the observed API order, STORAGE and HPA precede POD, and DCGM precedes JOBSET. The module defaults put STORAGE/HPA after STATEFULSET and append DCGM after JOBSET.
### Version (`gcluster --version`)
Source build at commit `8efc04f30` (`v1.103.0-15-g8efc04f30`). Terraform 1.16.1; Google provider 7.46.1; GKE 1.35.8-gke.1036000 in us-central1, REGULAR channel. The same ordering symptom occurred in an earlier run on GKE 1.35.7-gke.1222000.
### Blueprint
WIP agentgateway blueprint for https://github.com/GoogleCloudPlatform/cluster-toolkit/issues/6213. It inherits the shared module's monitoring defaults.
### Expanded Blueprint
No monitoring overrides: the shared module passes `monitoring_components`, appending DCGM when `enable_dcgm_monitoring` is true.
### Output and logs
```text
~ monitoring_config {
~ enable_components = [
"SYSTEM_COMPONENTS",
- "STORAGE",
- "HPA",
"POD",
# DAEMONSET and DEPLOYMENT unchanged
"STATEFULSET",
+ "STORAGE",
+ "HPA",
"CADVISOR",
"KUBELET",
- "DCGM",
"JOBSET",
+ "DCGM",
]
}
Plan: 0 to add, 1 to change, 0 to destroy.
```
### Execution environment
- macOS, arm64
- zsh
- Go 1.26.4
### Additional context
Related to #6213. The agentgateway deployment and real synchronous/streaming prediction tests pass. This issue is related to reapply stability in the shared GKE module.
A workaround was verified in a fresh live deployment for the agentgateway blueprint being added for #6213: explicitly provide monitoring components in the observed GKE order and disable DCGM for the initial CPU-only example. The broader fix should handle ordering for other blueprints and component selections, including DCGM, without ignoring actual monitoring changes. This may require a provider fix: the [provider schema](https://github.com/hashicorp/terraform-provider-google/blob/v7.46.1/google/services/container/resource_container_cluster.go) currently represents `enable_components` as `schema.TypeList`.
Retain the empty-plan integration check rather than suppressing monitoring changes with `ignore_changes`.
Contributor guide
Research direction
Start with the shared modules/scheduler/gke-cluster module and reproduce the issue using terraform plan -detailed-exitcode after reapplying an unchanged regional GKE cluster. Inspect the Google provider container cluster schema, where enable_components is represented as TypeList, and compare it with the API order shown. Done means an unchanged component set produces an empty plan while additions and removals still appear as changes.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- google-cloud, kubernetes, terraform
- Domain
- cloud, infrastructure
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 52/100