google / google/stellar-engine

[Bug] postgresql and app-engine blueprints require APIs the framework never enables (sqladmin.googleapis.com, appengine.googleapis.com) — fresh-tenant applies fail until the operator enables them manually

Open
#104 0 comments 0 reactions 1 assignee Claimed by @Calvin-Cheng1 View on GitHub
bug Level of Effort - Low Priority - Medium
Dominant language
HCL
Stars
49
Forks
20
Avg merge
3d 14h
Merged PRs (30d)
28

Description

## Bug Description

Stage 1 enables 22 APIs on each tenant main project (fast/stages-aw/1-resman/branch-tenants.tf, services lists at lines 183+ and 308+); neither sqladmin.googleapis.com nor appengine.googleapis.com is among them. A sweep of all 56 blueprints at 3728fc98 shows the PostgreSQL and App Engine blueprints are the only two that deploy into an existing Stage-1 tenant project, hard-require an API absent from that list, and contain no google_project_service resource to enable it. Specifically: postgresql's only service-related resource is google_project_service_identity (main.tf:24-27), which references the Cloud SQL service agent but does not enable the API — so google_sql_database_instance fails with 403 SERVICE_DISABLED, and the KMS grant to the (never actually provisioned) service agent misbehaves alongside it; the app-engine blueprint is a bare google_app_engine_application via modules/app-engine, with no enablement in either blueprint or module. This is an outlier pattern: the majority of blueprints enable their required services explicitly via google_project_service (e.g. cloud-run, cloud-functions, bigquery, datastore, firestore, gke, secret-manager, document-ai, cloud-ids, and ~20 more), others need only APIs Stage 1 already enables (compute, cloudkms, storage, pubsub, bigquery, container, …), and the rest create their own projects with their own service lists (vertex-mlops, gitlab). Both failures and the manual workaround are live-verified.

## Environment and Deployment Context
* **Stellar Engine Version/Commit:** `main` at commit `3728fc98`
* **Deployment Type:**
* [ ] US Region Restricted (e.g., Access Policy constraint)
* [ ] FedRAMP Medium
* [x] FedRAMP High
* [ ] DoD IL4
* [ ] DoD IL5
* [ ] Stand-alone / Custom
* **FAST Stage (if applicable):**
* [ ] Stage 0 (Bootstrap)
* [x] Stage 1 (Resource Management)
* [ ] Stage 2 (Network Creation)
* [ ] Stage 3 (Security and Audit)
* **Affected Component:** `fast/stages-aw/1-resman/branch-tenants.tf` (tenant services lists, lines 183+/308+ — zero occurrences of either API); `blueprints/il5/postgresql/main.tf` (`google_project_service` count: 0; `google_project_service_identity` at line 24); `blueprints/fedramp-high/app-engine/main.tf` + `modules/app-engine/main.tf` (`google_project_service` count: 0)
* **Terraform Version:** 1.15.7
* **GCP Provider Version:** `hashicorp/google` 6.50.0 (postgresql lock file; the failure is API-enablement and provider-version-independent)

## Steps to Reproduce
1. Deploy Stages 0–1 with a fresh tenant (services list as shipped, unmodified).
2. Deploy `blueprints/il5/postgresql` against the tenant's main project. The apply fails on `google_sql_database_instance` with 403 `SERVICE_DISABLED` (log below).
3. Run `gcloud services enable sqladmin.googleapis.com --project=` and re-apply — it now succeeds. The manual enable is the only intervention required, confirming root cause.
4. Repeat with `blueprints/fedramp-high/app-engine`: `google_app_engine_application` fails identically until `gcloud services enable appengine.googleapis.com` is run; after enablement the apply succeeds (`module.app-engine.google_app_engine_application.app` confirmed in state).
5. Code confirmation: `grep -c "sqladmin\|appengine" fast/stages-aw/1-resman/branch-tenants.tf` → 0; `grep -c 'resource "google_project_service"' blueprints/il5/postgresql/main.tf blueprints/fedramp-high/app-engine/main.tf modules/app-engine/main.tf` → 0 in each.

## Expected Behavior
Blueprints shipped for tenant projects can apply against a fresh tenant: either Stage 1 enables the APIs the shipped workload blueprints require (as it already does for e.g. `cloudkms.googleapis.com`), or these two blueprints enable their own APIs via `google_project_service`, as the majority of sibling blueprints already do.

## Actual Behavior
Every fresh-tenant deployment of either blueprint hard-fails on apply and cannot self-heal; the operator must discover and run a manual `gcloud services enable` that no README documents.

## Relevant Logs and Errors
Captured from a live failed apply (postgresql, `TF_LOG` debug; project number redacted):
```
google_project_service_identity.cloudsql_sa: Refreshing state...
[id=projects//services/sqladmin.googleapis.com]
...
Error: Error, failed to create instance report: googleapi: Error 403:
Cloud SQL Admin API has not been used in project before or it
is disabled. Enable it by visiting
https://console.developers.google.com/apis/api/sqladmin.googleapis.com/overview?project=
then retry. ...
"reason": "SERVICE_DISABLED"
"service": "sqladmin.googleapis.com"
```
Note the misleading shape: `google_project_service_identity.cloudsql_sa` refreshes without error while the instance create fails — nothing points to a missing enablement in the framework itself as the root cause.

## Additional Context
Blocks primary workload deliverables (Cloud SQL, App Engine) on every fresh tenant. `blueprints/fedramp-high/postgresql` is a symlink to `blueprints/il5/postgresql`, so FRH and IL5 consume identical code and are equally affected. Proposed fix (either or both): add `sqladmin.googleapis.com` and `appengine.googleapis.com` to Stage 1's tenant services lists, and/or add `google_project_service` resources to the two blueprints, matching the majority-blueprint pattern. Note that enabling the API does not by itself make impersonated deploys work — see the companion documentation issue on deployer permissions and `user_project_override`: {companion-issue link}. Related (not duplicate): #47 proposes centralizing bootstrap-level prerequisite/API documentation; this issue is a code gap at the tenant/blueprint level.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.