GoogleCloudPlatform / GoogleCloudPlatform/cluster-toolkit

GKE monitoring component ordering causes persistent Terraform drift

Open
#6,331 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
361
Forks
322
Avg merge
2d 21h
Merged PRs (30d)
115

Description

### Describe the bug

The shared `modules/scheduler/gke-cluster` module produces a persistent Terraform diff when GKE returns `monitoring_config.enable_components` in a different order from the configured list. Reapplying succeeds but does not remove the diff, causing unnecessary cluster updates and failing integration tests that require an empty plan.

### Steps to reproduce

1. Deploy a regional GKE Standard cluster using the module's default `monitoring_components` and `enable_dcgm_monitoring` settings.
2. Reapply the deployment without changing its configuration.
3. Run `terraform plan -detailed-exitcode`.

### Expected behavior

An unchanged set of monitoring components should produce an empty plan (exit 0), while actual additions and removals remain managed by Terraform.

### Actual behavior

The plan exits 2 and proposes an in-place cluster update. The component membership is unchanged, only ordering differs. In the observed API order, STORAGE and HPA precede POD, and DCGM precedes JOBSET. The module defaults put STORAGE/HPA after STATEFULSET and append DCGM after JOBSET.

### Version (`gcluster --version`)

Source build at commit `8efc04f30` (`v1.103.0-15-g8efc04f30`). Terraform 1.16.1; Google provider 7.46.1; GKE 1.35.8-gke.1036000 in us-central1, REGULAR channel. The same ordering symptom occurred in an earlier run on GKE 1.35.7-gke.1222000.

### Blueprint

WIP agentgateway blueprint for https://github.com/GoogleCloudPlatform/cluster-toolkit/issues/6213. It inherits the shared module's monitoring defaults.

### Expanded Blueprint

No monitoring overrides: the shared module passes `monitoring_components`, appending DCGM when `enable_dcgm_monitoring` is true.

### Output and logs

```text
~ monitoring_config {
~ enable_components = [
"SYSTEM_COMPONENTS",
- "STORAGE",
- "HPA",
"POD",
# DAEMONSET and DEPLOYMENT unchanged
"STATEFULSET",
+ "STORAGE",
+ "HPA",
"CADVISOR",
"KUBELET",
- "DCGM",
"JOBSET",
+ "DCGM",
]
}
Plan: 0 to add, 1 to change, 0 to destroy.
```

### Execution environment

- macOS, arm64
- zsh
- Go 1.26.4

### Additional context

Related to #6213. The agentgateway deployment and real synchronous/streaming prediction tests pass. This issue is related to reapply stability in the shared GKE module.

A workaround was verified in a fresh live deployment for the agentgateway blueprint being added for #6213: explicitly provide monitoring components in the observed GKE order and disable DCGM for the initial CPU-only example. The broader fix should handle ordering for other blueprints and component selections, including DCGM, without ignoring actual monitoring changes. This may require a provider fix: the [provider schema](https://github.com/hashicorp/terraform-provider-google/blob/v7.46.1/google/services/container/resource_container_cluster.go) currently represents `enable_components` as `schema.TypeList`.

Retain the empty-plan integration check rather than suppressing monitoring changes with `ignore_changes`.

Contributor guide

Open the contributing guide

Research direction

Start with the shared modules/scheduler/gke-cluster module and reproduce the issue using terraform plan -detailed-exitcode after reapplying an unchanged regional GKE cluster. Inspect the Google provider container cluster schema, where enable_components is represented as TypeList, and compare it with the API order shown. Done means an unchanged component set produces an empty plan while additions and removals still appear as changes.

Written by the indexing model from the issue text.

Assessment

Tech stack
google-cloud, kubernetes, terraform
Domain
cloud, infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
52/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.