google / google/stellar-engine

[Bug] gemini-enterprise: usage-audit logging half-applies and nothing notices — the engine is left observabilityEnabled without sensitiveLoggingEnabled, so every prompt and response logs as <elided>

Open
#182 0 comments 0 reactions 0 assignees View on GitHub
bug gemini for government Level of Effort - Low Priority - Medium security
Dominant language
HCL
Stars
49
Forks
20
Avg merge
3d 14h
Merged PRs (30d)
28

Description

## Bug Description
`deploy.sh` offers to "enable Gemini Enterprise Usage Audit logs (conversation logging)" and warns that doing so "will write user queries, model thinking, and model responses to Cloud Logging". When the operator accepts, the post-apply step PATCHes the engine with `{"observabilityConfig": {"observabilityEnabled": true, "sensitiveLoggingEnabled": true}}`.

On a completed FedRAMP-High deployment the engine is instead left in a state the script never sends: `observabilityEnabled: true` with `sensitiveLoggingEnabled` absent (i.e. false). Per the Discovery Engine schema, `sensitiveLoggingEnabled` "includes customer core content (e.g. prompts, responses). If `false`, will sanitize all sensitive fields." The result is audit logging that looks enabled and captures none of the content it was enabled for.

Because the PATCH is issued as `curl -s -o /dev/null` with no status or body check (deploy.sh :2524-2540 — the same pattern is used for the assistant-compliance PATCH immediately after), a rejected, ignored or partially-applied field produces no output, no warning and no non-zero exit. Nothing later in the blueprint or the tooling reads `observabilityConfig` back.

## Environment and Deployment Context
* **Stellar Engine Version/Commit:** `main` at commit `f64ce6cd` (re-verified 2026-08-10)
* **Deployment Type:**
* [ ] US Region Restricted (e.g., Access Policy constraint)
* [ ] FedRAMP Medium
* [x] FedRAMP High
* [ ] FedRAMP Moderate
* [ ] DoD IL4
* [ ] DoD IL5
* [ ] Stand-alone / Custom
* **FAST Stage (if applicable):** N/A — this is a blueprint, not a FAST stage
* [ ] Stage 0 (Bootstrap)
* [ ] Stage 1 (Resource Management)
* [ ] Stage 2 (Network Creation)
* [ ] Stage 3 (Security and Audit)
* **Affected Component:** `blueprints/fedramp-high/gemini-enterprise/deploy.sh` — post-apply observability PATCH
* **Terraform Version:** `1.12.2` (pinned by `deploy.sh` via tfenv; the stage declares `required_version >= 1.7.4`)
* **GCP Provider Version:** `hashicorp/google >= 6.21.0` (stage-0 declared constraint)
## Steps to Reproduce
1. Run `deploy.sh`, answer **y** to the Usage Audit logs prompt, and complete the apply.
2. `GET .../engines/{ENGINE_ID}` and read `observabilityConfig`.
3. Run assistant and search queries against the app, then read the `discoveryengine.googleapis.com/gemini_enterprise_user_activity`, `gen_ai.user.message` and `gen_ai.choice` logs.

## Expected Behavior
Either the engine reads `{observabilityEnabled: true, sensitiveLoggingEnabled: true}` and prompts and responses appear in the logs, or the deployment reports that the setting could not be applied.

## Actual Behavior
`observabilityConfig` reads `{"observabilityEnabled": true}` only. In the logs (44 user-activity entries, 164 `gen_ai.user.message`, 200+ `gen_ai.choice` sampled):
- **sanitized to ``** — `gen_ai.user.message.content` (assistant prompt, 164/164), `gen_ai.choice.content` (model answer, 200/200), `serviceTextReply`, grounding-reference `content` / `title` / `mimeType` / `domain`, **and `userIamPrincipal` (44/44)**;
- **retained in the clear** — `request.query` on `SearchService.Search` (search-box text), `request.userEvent.searchInfo.searchQuery`, grounding-reference `uri`, method names, timestamps and resource names;
- StreamAssist entries carry `request.query: {}` — the assistant prompt is not present at all.

Re-issuing exactly the same PATCH by hand **succeeds** (HTTP 200, both flags echoed, and a follow-up GET confirms `sensitiveLoggingEnabled: true` persisted). So the API accepts the field; the deployment nevertheless ended up without it, and no code path in the blueprint produces the mixed state that was observed (`deploy.sh` sends both flags true or both false; `gem4gov update-compliance` patches the engine with `updateMask=features` only; `gemini-stage-0` Terraform sets no observability config). Root cause is therefore undetermined from source alone — which is itself the point of the report: the silent write makes it undiagnosable after the fact.

## Relevant Logs and Errors
Observed live 2026-08-06. The engine read back after a completed deployment:

```json
{ "observabilityConfig": { "observabilityEnabled": true } }
```

`sensitiveLoggingEnabled` is absent. The PATCH that should have set it is issued as `curl -s -o /dev/null` with no status or body check, so nothing reported a failure:

```bash
# deploy.sh:2526-2531
curl -s -o /dev/null -X PATCH \
-H "Authorization: Bearer ${ACCESS_TOKEN}" \
... \
"https://us-discoveryengine.googleapis.com/v1alpha/projects/${PROJECT_ID}/locations/us/collections/default_collection/engines/${ENG_ID}?updateMask=observabilityConfig" \
-d '{"observabilityConfig": {"observabilityEnabled": true, "sensitiveLoggingEnabled": true}}'
```

## Additional Context
Impact is compliance-facing, not cosmetic: an operator who accepted the prompt will tell an auditor that conversation logging is on. What is actually retained is search-box query text with the acting principal removed — while the assistant conversation, every response, and the identity of the user are all withheld.

Minimal fix: capture the PATCH status/body (drop `-o /dev/null`, check the HTTP code and the echoed `observabilityConfig`), fail or warn loudly on a mismatch, and read the config back once after the apply. The same treatment is warranted for the assistant-compliance PATCH that follows it.

Contributor guide

Open the contributing guide

Research direction

Start in blueprints/fedramp-high/gemini-enterprise/deploy.sh around lines 2524-2540 and reproduce the Usage Audit logs path by answering y. Inspect the PATCH response and then GET the engine's observabilityConfig; done means failures or mismatches are reported clearly and the expected settings are confirmed, with the adjacent assistant-compliance PATCH receiving the same treatment.

Written by the indexing model from the issue text.

Assessment

Tech stack
gcp, shell
Domain
cloud, devops, observability
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
68/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.