google / google/stellar-engine

[Bug] Brownfield discovery offers the wrong Terraform state bucket: it looks for gs://<prefix>-tn-<env>-<tenant>-0, but a tenant's own stage state lives in gs://<prefix>-<env>-<tenant>-iac-0

Open
#230 0 comments 0 reactions 0 assignees View on GitHub
bug gemini for government Level of Effort - Low Priority - High
Dominant language
HCL
Stars
49
Forks
20
Avg merge
3d 14h
Merged PRs (30d)
28

Description

## Bug Description

> **Both names in this report are Cloud Storage buckets, not projects.** They are easy to misread,
> because a bucket name here differs from a project name by one word: the bucket
> `---iac-0` lives *inside* the project `---iac-core-0`.
> No project carries `-tn-` in its name at all.

PR #217 changed the state-bucket lookup in `discover_infrastructure()`:

```
# deploy.sh:584 (main)
POTENTIAL_BUCKET="${PREFIX}-tn-${ENVIRONMENT}-${TENANT}-0"

# previously (v2.13.0 and every release before it)
POTENTIAL_BUCKET="${PREFIX}-${ENVIRONMENT}-${TENANT}-iac-0"
```

The stated reason was that `1-resman` creates the tenant state bucket as `-tn---0`
"so the bucket was never found and a second one was created."

**`1-resman` creates three buckets per tenant environment, with three different roles, and the stage
picked the wrong one.** This is not a guess about intent — `1-resman/outputs-tenants.tf` names them:

```hcl
# outputs-tenants.tf — local.tenant_tfvars
automation = {
core_bucket = module.tenant-core-gcs[k].name # -tn---0
outputs_bucket = module.tenant-self-iac-gcs-outputs[k].name # ---iac-outputs-0
state_bucket = module.tenant-self-iac-gcs-states[k].name # ---iac-0
}
```

**The bucket upstream itself calls `state_bucket` is `---iac-0` — the one the
lookup used to find, and the one PR #217 moved away from.**

The same file builds two distinct backend configurations from the same template, and they are not
interchangeable because they run as different identities:

| Local | Backend bucket | Runs as | Lives in |
|---|---|---|---|
| `tenant_core_providers` | `core_bucket` = `-tn---0` | `tenant-core-sa` | the **organization** automation project |
| `tenant_self_providers` | `state_bucket` = `---iac-0` | `tenant-self-iac-sa` | the **tenant's own** IaC core project |

A Gemini Enterprise deployment is tenant-side work, so its state belongs in `state_bucket` — which is
where the pre-#217 lookup pointed and where every existing deployment has it. `core_bucket` is the
backend for the organization-side configuration that manages the tenant; on a landing zone where
nobody has applied that configuration it is simply empty, which is exactly what makes the wrong
choice silent.

The consequence is not a missing bucket. It is a **silently different backend**: on a landing zone
where both buckets exist, the wizard now binds stage 0 to the organization-level bucket, which holds
no Gemini state, so Terraform sees an empty state and plans to create a CmekConfig, a reserved VIP, a
Discovery Engine app and data stores that already exist in the project.

**Nothing changed on the `1-resman` side — this is a regression in `deploy.sh` alone.** The three
buckets, their names, and the `core_bucket` / `state_bucket` labels in `outputs-tenants.tf` are
byte-identical at `v2.9.0`, `v2.11.0`, `v2.12.0`, `v2.13.0`, `v3.0.0` and `main`; `tenant-core-gcs`
has existed since 2024-09-16 (`a39c7ff4`). So the premise behind the change — that `1-resman` names
the tenant state bucket `-tn---0` — was never true at any release, and the
lookup it replaced had been correct for the life of the stage.

The hyphenated-tenant half of PR #217 (`ten-1` no longer parsed as `ten`) is a genuine fix and is not
in question.

## Environment and Deployment Context
* **Stellar Engine Version/Commit:** `main` @ `6d7d08c0` (2026-09-09). Introduced by PR #217 (`60a81387`, merged 2026-09-05). `v2.13.0` (`8f5b67a6`) and `v3.0.0` (`f64ce6cd`) both use the previous name.
* **Deployment Type:**
* [ ] US Region Restricted (e.g., Access Policy constraint)
* [ ] FedRAMP Moderate
* [x] FedRAMP High
* [ ] DoD IL4
* [ ] DoD IL5
* [ ] Stand-alone / Custom
* **FAST Stage (if applicable):** the `gemini-enterprise` blueprint's brownfield discovery, against a landing zone built by Stages 0-3
* [ ] Stage 0 (Bootstrap)
* [x] Stage 1 (Resource Management)
* [ ] Stage 2 (Networking)
* [ ] Stage 3 (Security)

## Steps to Reproduce
1. Deploy Stages 0-1 with any prefix and a tenant (for example `g4g`), so that `1-resman` creates both `-tn---0` and `---iac-0`.
2. Deploy the Gemini Enterprise blueprint stage 0 from a release at or before `v3.0.0`. Its state is written to `gs://---iac-0/`.
3. Update the checkout to `main` and re-run `./deploy.sh` in brownfield mode against the same project.
4. Read the bucket it reports at "Checking for Terraform State Bucket", and the backend it writes.

## Expected Behavior
Discovery finds the bucket that holds the deployment's existing state, so a re-run is a no-op plan.

## Actual Behavior
Discovery reports `Found Terraform State Bucket: -tn---0` and offers it as the
default of the state-bucket prompt. **Nothing rejects it:** the bucket exists, so no fallback fires,
and it passes the CMEK check at `deploy.sh:690-700` because `1-resman` encrypts *both* tenant buckets
with the same key. An operator who accepts the offered default — the documented path — binds the
stage to a bucket holding no state, and the plan proposes to create the whole stage a second time on
top of live resources.

Observed on a FedRAMP High landing zone, prefix `alaska`, tenants `g4g` and `report`, all read-only:

```
$ gcloud storage ls --recursive 'gs://alaska-prod-g4g-iac-0/**'
gs://alaska-prod-g4g-iac-0/terraform/state/stage-0/default.tfstate

$ gcloud storage ls --recursive 'gs://alaska-int-g4g-iac-0/**'
gs://alaska-int-g4g-iac-0/terraform/state/stage-0/default.tfstate
gs://alaska-int-g4g-iac-0/terraform/state/stage-1/default.tfstate

$ gcloud storage ls --recursive 'gs://alaska-tn-prod-g4g-0/**'
ERROR: (gcloud.storage.ls) One or more URLs matched no objects.

$ gcloud storage ls --recursive 'gs://alaska-tn-int-g4g-0/**'
ERROR: (gcloud.storage.ls) One or more URLs matched no objects.
```

The `-iac-0` buckets hold the blueprint's own state, at a path that names the Gemini stages
explicitly. The `-tn-` buckets — the ones the new code selects — are **completely empty**.

Both are CMEK-encrypted with the same key and both are versioned, so nothing downstream distinguishes
them either:

```
$ gcloud storage buckets describe gs://alaska-prod-g4g-iac-0 --format='value(default_kms_key,versioning_enabled)'
projects/alaska-prod-g4g-iac-core-0/locations/us-west1/keyRings/Prod-g4g-keyring/cryptoKeys/gcs True

$ gcloud storage buckets describe gs://alaska-tn-prod-g4g-0 --format='value(default_kms_key,versioning_enabled)'
projects/alaska-prod-g4g-iac-core-0/locations/us-west1/keyRings/Prod-g4g-keyring/cryptoKeys/gcs True
```

And the two buckets live in different projects, which is the clearest statement of the difference:
`---iac-0` is in the tenant's own IaC project
(`alaska-prod-g4g-iac-core-0`), while `-tn---0` is in the organization
automation project (`alaska-prod-iac-core-0`) alongside the resman buckets.

## Relevant Logs and Errors
No error. The run looks clean until the plan output is read.

## Additional Context
* Suggested fix: look for `state_bucket` (`---iac-0`) rather than `core_bucket`; if it is absent, fall back to `core_bucket`, and if both exist and both hold state, prompt rather than choose. A bucket that exists but contains no `*.tfstate` should not be accepted silently.
* Worth guarding generally: the wizard treats "bucket exists" as "state found". Checking for an object under the expected prefix would have caught this **and** the original problem #217 set out to fix — one `gcloud storage ls` on the bucket would distinguish them.
* For completeness, there is a third name in play: if the CMEK check does reject the chosen bucket, `deploy.sh:718` falls back to creating `---tfstate-0`, which matches neither of the two `1-resman` produces. That path is not what happens here — both tenant buckets carry the key — but it means a rejected bucket also does not lead back to the right one.
* The remaining brownfield assumption the PR author set aside — the derived `--keyring` in the tenant IaC project, which `1-resman` does not create — is reported separately as #132 and #106.

Contributor guide

Open the contributing guide

Research direction

Start in deploy.sh at discover_infrastructure(), especially the bucket lookup around line 584, the CMEK check at lines 690-700, and the fallback near line 718. Reproduce brownfield discovery with the documented gcloud storage listings and verify it selects the tenant state bucket containing the expected *.tfstate object. Done means rerunning the existing deployment produces a no-op plan instead of proposing duplicate resources.

Written by the indexing model from the issue text.

Assessment

Tech stack
gcp, shell, terraform
Domain
cloud, devops, infrastructure
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
74/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.