GoogleCloudPlatform / GoogleCloudPlatform/BigQuery-Agent-Analytics-SDK

Admin agent skill: let Claude Code / Codex drive the full rollout — install → deployed, validated, finalized BQ logging dataset

Open
#350 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
47
Forks
21
Avg merge
2d 13h
Merged PRs (30d)
33

Description

## Why

The #316 → #344 arc produced a complete admin stack and a rehearsed demo — and, as a byproduct, a recorded transcript of every mistake an agent can make deploying it (nonexistent gcloud subcommands, org-policy blocks, per-product landing false-greens, enum drift, unreliable shutdown flushes, unverifiable teardowns). An **agent skill** distills that operational knowledge so an enterprise admin can point their own Claude Code or Codex at a GCP project and get from `pipx install` to a finalized, validated telemetry dataset — with the judgment points and safety rails we learned live built in.

Depends on **#349** (`bootstrap --preflight` and `bqaa-otel teardown` must exist as subcommands first).

## Core design principle: skill orchestrates, CLI executes

No deployment logic in the skill's bash. Mechanics live in `bqaa-otel` (preflight/bootstrap/verify/teardown, all live-proven); the skill is the **conversational conductor** — what to run next, how to interpret failure, when to stop and ask the human. This keeps the skill thin and product-agnostic: the same content drives Claude Code and Codex because the commands do the heavy lifting, and the skill versions with the CLI it drives (`requires bqaa-otel >= 0.2.0` — skill/CLI drift is the failure mode to design against).

## Phase structure (mirrors the proven live sequence)

```
0. Install pipx install bigquery-agent-analytics-tracing==X → version check
1. Interview project / dataset / region / signals / privacy tier
2. Preflight bqaa-otel bootstrap --preflight → interpret via failure map
3. Deploy show plan (dry-run) → explicit human confirm → --execute
4. Enable distribute the two generated artifacts (the branchy part)
5. Validate real session → per-product landing gates → verify --smoke
6. Finalize retention policy, analyst IAM, saved views, DLQ/freshness alerts
7. Handoff inventory recorded for teardown + evidence summary doc
```

Two phases carry most of the authoring effort:

- **Phase 2 — the failure→fix map.** Learned live: org policy blocks `allUsers` → name the constraint and the ask ("get a policy exception for `iam.allowedPolicyMemberDomains`"); missing `iam.serviceAccounts.actAs` → the exact role to request; billing invisible → who to contact. The agent should emit the *remediation sentence*, not the raw error. Seed content: the demo OPERATOR.md troubleshooting table (built from real failures).
- **Phase 4 — where agents improvise dangerously without guidance.** Branches by product and channel: Claude managed settings via admin console (**human-only paste** — the skill must say "I cannot do this step; here is exactly what to paste") vs MDM; Codex `config.toml` via managed dotfiles with the embedded-token warning. This phase is mostly instructions *to the human*, and the skill is explicit about that boundary.

## Safety rails (each one is a live scar)

- **The token never enters the chat.** Secret Manager references only; the skill forbids echoing generated Codex configs (they contain the literal bearer token).
- **Per-product landing gates, never aggregate.** "It works" where only one product landed is the exact false-green caught in #345 review round 3.
- **Verification is existence checks, not exit codes** (the rule the #346 teardown review forced — probe failures must never masquerade as "gone").
- **Privacy is a one-way door with a human key.** The skill never constructs the replay acknowledgement flag itself; the human must type it. Mirrors the CLI's exit-2 gate at the conversational layer.
- **Dry-run before anything destructive; teardown only via the recorded inventory** with allowlist-guarded names.

## Distribution

- **Claude Code**: ship the skill inside the plugin tarball that `release-tracing.yml` already builds — versioned with the CLI.
- **Codex**: emit the same content as an AGENTS.md playbook. Candidate mechanism: `bqaa-otel init-skill --agent claude-code|codex` so one source generates both — symmetric with how the product already generates per-product config artifacts.

## Acceptance: the demo is the eval

The #344 rehearsal is the golden transcript, so the skill's acceptance test is concrete:

> Fresh GCP project, clean machine, an agent given only the skill and a project ID completes phases 0–7 **without human help except the designed-in confirmation points** (deploy confirm, artifact paste, privacy-tier choice).

- [ ] Skill source authored (phases, failure→fix map, safety rails), with the skill/CLI version pin
- [ ] `init-skill` (or equivalent) emits Claude Code skill + Codex AGENTS.md from one source
- [ ] Wired into the release plugin tarball
- [ ] Eval run recorded: agent-driven rollout on a fresh dataset, graded against the rehearsal transcript — evidence posted here
- [ ] Negative tests: agent correctly *refuses* to echo tokens, to enable replay without the human ack, and to tear down without a dry-run shown

Contributor guide

Open the contributing guide

Research direction

Start with the demo OPERATOR.md troubleshooting table and the existing release-tracing.yml plugin packaging. Confirm the #349 subcommands and their versioning, then design init-skill to emit the Claude Code skill and Codex AGENTS.md from one source. Done means the rehearsal-based eval and negative tests pass, with evidence recorded for the rollout and safety rails.

Written by the indexing model from the issue text.

Assessment

Tech stack
google-cloud, python
Domain
cli, cloud, devops, observability, release, testing
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.