Move tracing producers (OpenAI Agents SDK, Codex CLI, Claude Code plugin) into this repo and publish PyPI + plugin artifacts

## Summary

Move the tracing producer surfaces from
[`Erroration2022/bigquery_agent_analytics_skill`](https://github.com/Erroration2022/bigquery_agent_analytics_skill/tree/main/plugins/bigquery-agent-analytics-tracing)
into this repo as first-class BQAA producers, and publish them through the
install channels their users actually use:

- Python package: `bigquery-agent-analytics-tracing`
- Claude Code plugin marketplace entry: `bigquery-agent-analytics-tracing`
- Codex CLI wrapper console script: `bqaa-codex`

These producers already emit rows compatible with the canonical BQAA
`agent_events` shape consumed by this SDK. The move should preserve row shape
and producer labels so existing queries, dashboards, and SDK consumers keep
working.

## What exists today in the source repo

Verified against the current code under
`plugins/bigquery-agent-analytics-tracing/`:

| Surface | Current implementation | Notes for port |
|---|---|---|
| Shared Python writer | `sdk/python/bqaa_tracing.py` | Defines `BQAAConfig`, `BigQueryAgentAnalyticsLogger`, `bq_schema()`, Claude hook adapter, dry-run, direct-write, local spool, writer attribution, and state locking. |
| Async drainer | `sdk/python/bqaa_drain.py` plus `scripts/bqaa_drain.py` wrapper | Uses a pidfile `flock`, batches spool files, writes through BigQuery Storage Write API with `pyarrow`, falls back to `insert_rows_json`, retries transient failures, and dead-letters failed rows. |
| OpenAI Agents SDK | `sdk/python/bqaa_openai_agents.py` | Optional `openai-agents` dependency. Implements `BQAAOpenAIAgentsProcessor` and `add_bqaa_trace_processor()`. Maps `generation` spans to `LLM_REQUEST`/`LLM_RESPONSE`, `function` spans to `TOOL_STARTING`/`TOOL_COMPLETED`, and other spans to `STATE_DELTA`. |
| Codex CLI | `sdk/python/bqaa_codex.py` plus `scripts/bqaa_codex.py` wrapper | Wraps `codex exec --json` only. Captures argv/stdin prompt, forwards Codex output to stdout, maps JSONL stream events to BQAA rows, records raw Codex event/item types for drift debugging. Interactive Codex TUI is not captured today. |
| Claude Code plugin | `.claude-plugin/plugin.json`, `hooks/*.sh`, `scripts/bqaa_hook.py`, `commands/bqaa-setup.md`, `skills/bqaa-setup/SKILL.md` | Native hook plugin. Hooks shell through `hooks/common.sh` to `scripts/bqaa_hook.py`, which inserts `sdk/python` on `sys.path` and calls the shared hook adapter. |
| Claude Agent SDK | Documented in `USER_GUIDE.md` | Not a separate Python helper yet. The documented path loads the same Claude Code plugin into the SDK-spawned `claude` subprocess via `ClaudeAgentOptions.plugins` or `extra_args={"plugin-dir": ...}` and passes `BQAA_*` env explicitly. |
| Bootstrap | `scripts/setup_gcp_prereqs.py`, `commands/bqaa-setup.md`, `skills/bqaa-setup/SKILL.md` | Dry-run first, explicit approval before `--execute`, separate ADC vs gcloud CLI auth checks, dataset/table/API/IAM setup, and consumer-specific env handoff. |
| Tests / smokes | `scripts/test_codex_wrapper.py`, `scripts/test_setup_gcp_prereqs.py`, `scripts/e2e_bigquery_smoke.py`, `scripts/e2e_openai_agents_smoke.py` | Unit tests are script-style today. BigQuery smokes are real round-trip tests and should stay gated/opt-in in CI. |
| Codex plugin metadata | `.codex-plugin/plugin.json`, `codex-marketplace.example.json` | Useful packaging metadata, but Codex plugins package commands/prompts. They do not currently provide Claude-style runtime hooks for Codex TUI sessions. |

## Target shape

This repo currently ships the consumption SDK as one Hatch package:
`bigquery-agent-analytics`, with packages under `src/bigquery_agent_analytics`
and `src/bigquery_ontology`. The tracing producers should stay separate from
that consumption wheel so producer installs do not pull evaluator, BigFrames,
ontology, or ADK consumption dependencies.

Proposed layout:

```text
producers/
pyproject.toml # package: bigquery-agent-analytics-tracing
src/bigquery_agent_analytics_tracing/
__init__.py
config.py # from BQAAConfig
logger.py # from BigQueryAgentAnalyticsLogger
schema.py # from bq_schema()
drain.py # from bqaa_drain.py
claude_code.py # ClaudeHookBQAAAdapter + hook entry main()
codex.py # from bqaa_codex.py
openai_agents.py # from bqaa_openai_agents.py
setup_gcp_prereqs.py # from setup_gcp_prereqs.py
claude-code-plugin/
.claude-plugin/plugin.json
hooks/
scripts/ # thin wrappers or vendored package files
commands/
skills/
codex-plugin/
.codex-plugin/plugin.json
codex-marketplace.example.json
tests/
```

Alternative if maintainers prefer one root package only:
`src/bigquery_agent_analytics_tracing/` at repo root with a second Hatch build
target. The key requirement is that `bigquery-agent-analytics-tracing` remains
a separate PyPI distribution from `bigquery-agent-analytics`.

## Proposed Python package

PyPI package: `bigquery-agent-analytics-tracing`

Base install:

```bash
pip install bigquery-agent-analytics-tracing
```

Extras:

```bash
pip install "bigquery-agent-analytics-tracing[storage-write]" # google-cloud-bigquery-storage + pyarrow
pip install "bigquery-agent-analytics-tracing[openai-agents]" # openai-agents
pip install "bigquery-agent-analytics-tracing[claude-sdk]" # claude-agent-sdk examples/helper, if we add one
```

The `bqaa-codex` console script ships in the base install. It has no extra
Python runtime dep — the `codex` binary itself is required at runtime, and the
wrapper exits with a clear error if it is missing. Therefore there is no
`[codex]` extra (an extras tag that installs nothing would be cosmetic and
misleading).

Required base dependency should include `google-cloud-bigquery`, since both
the direct writer and fallback drainer path need it. `google-cloud-bigquery-storage`
and `pyarrow` should be optional because the current drainer already falls back
when those imports are unavailable.

Console scripts:

```toml
bqaa-codex = "bigquery_agent_analytics_tracing.codex:main"
bqaa-drain = "bigquery_agent_analytics_tracing.drain:main"
bqaa-setup = "bigquery_agent_analytics_tracing.setup_gcp_prereqs:main"
```

Public Python imports:

```python
from bigquery_agent_analytics_tracing import BigQueryAgentAnalyticsLogger
from bigquery_agent_analytics_tracing.openai_agents import add_bqaa_trace_processor
```

The port should avoid preserving the current flat module names
(`bqaa_tracing`, `bqaa_drain`, `bqaa_openai_agents`) as the only public API.
Compatibility shims are fine, but the new package should expose stable package
imports.

## Claude Code plugin packaging

Submit the Claude Code plugin as `bigquery-agent-analytics-tracing`.

Important packaging detail: the plugin should work without a separate `pip
install` step. Today the hook path is:

```text
hooks/*.sh -> hooks/common.sh -> scripts/bqaa_hook.py -> sdk/python on sys.path
```

After the port, the marketplace artifact should either:

- vendor the minimal tracing package files into the plugin artifact, or
- ship thin wrappers that import from a bundled local package directory.

Relying on a globally installed `bigquery-agent-analytics-tracing` wheel would
make marketplace install fragile.

**Caveat — "no separate install of the tracing wheel" is not the same as
"no Python install step."** The plugin still requires `BQAA_PYTHON` to have
`google-cloud-bigquery` available, and optionally
`google-cloud-bigquery-storage` + `pyarrow` for the Storage Write path. The
`/bqaa-setup` flow should detect missing runtime deps and surface a single
`pip install ...` line rather than letting the hook crash on first import.

The plugin includes:

- native Claude Code hooks for `SessionStart`, `UserPromptSubmit`,
`PreToolUse`, `PostToolUse`, `Stop`, `SubagentStop`, `Notification`,
`PermissionRequest`, and `SessionEnd`
- `/bqaa-setup` slash command and setup skill
- consumer-specific setup guidance for Claude Code TUI, Claude Agent SDK, and
Codex CLI

## Codex scope

Current support is a wrapper around `codex exec --json`, not a general Codex
runtime hook:

```bash
bqaa-codex --sandbox read-only "summarize this repo"
```

The wrapper should remain documented as `codex exec` only. Interactive Codex
TUI sessions are out of scope unless Codex adds a runtime hook or trace export
surface.

The existing `.codex-plugin/plugin.json` and `codex-marketplace.example.json`
can move with this work, but they should not be described as tracing the Codex
TUI. Codex plugin packaging is useful for commands/setup prompts, while actual
trace capture comes from the `bqaa-codex` wrapper.

## Claude Agent SDK scope

Current support is "load the Claude Code plugin into the SDK-spawned `claude`
process" rather than a separate SDK adapter:

```python
from claude_agent_sdk import ClaudeAgentOptions, ClaudeSDKClient

options = ClaudeAgentOptions(
cwd="/your/project/path",
env={
"BQAA_PROJECT_ID": "your-project",
"BQAA_DATASET": "agent_analytics",
"BQAA_TABLE": "agent_events",
"BQAA_LOCATION": "US",
"BQAA_AGENT_NAME": "claude-agent-sdk",
},
plugins=[{"type": "local", "path": "/path/to/bigquery-agent-analytics-tracing"}],
)
```

If we want a `claude-sdk` extra, it should add a small helper only after we
decide the API. Until then, treat Claude Agent SDK as docs plus plugin
packaging, not a separate producer implementation.

## Compatibility requirements

- Keep the table schema compatible with ADK BQAA `agent_events`:
`timestamp`, `event_type`, `agent`, `session_id`, `invocation_id`,
`user_id`, `trace_id`, `span_id`, `parent_span_id`, `content`,
`content_parts`, `attributes`, `latency_ms`, `status`, `error_message`,
`is_truncated`.
- Preserve existing producer defaults unless there is a deliberate migration:
- Claude Code default agent: `claude-code`
- Claude Agent SDK documented override: `claude-agent-sdk`
- Codex wrapper default agent: `codex-cli`
- OpenAI Agents default agent: `openai-agents`
- Preserve row-level writer attribution in `attributes.writer`:
`plugin`, `version`, `label`, `agent`, `mode`.
- Preserve source tags used by current docs and adoption queries:
- Codex: `attributes.source = "codex_cli"`
- OpenAI Agents: `source/session_metadata.source = "openai_agents_sdk"`
- Claude Code: existing `custom_tags.assistant = "claude_code"` behavior
- Keep local spool plus background drain as the default write mode.
- Keep `BQAA_DIRECT_WRITE=true` and `BQAA_DRY_RUN=true`.
- Keep `BQAA_PYTHON`, `BQAA_CODEX_BIN`, `BQAA_WRITER_LABEL`, `BQAA_SPOOL_DIR`,
`BQAA_STATE_DIR`, `BQAA_TRACE_ENABLED`, and setup-related env behavior.

## Alignment with ADK 2.0

Issue #190 is rolling an ADK 2.0 schema through this repo, with sub-issues
#194–#221 covering the producer/consumer contract. Parallel ADK producer
additions tracked under #190 include:

- `attributes.adk.schema_version` on every enriched row
- `attributes.adk.source_event_id` as the join key for rows with an originating
ADK `Event`
- `attributes.adk.scope` (`{id, kind}` or `null`) for workflow- and node-scoped
events
- New ADK-only event types such as `AGENT_TRANSFER`, `EVENT_COMPACTION`,
`AGENT_STATE_CHECKPOINT`

The non-ADK producers being moved here (OpenAI Agents SDK, Codex CLI, Claude
Code, Claude Agent SDK loading path) have no originating ADK `Event` and
cannot populate `attributes.adk.source_event_id`. The v0.1 release should:

1. Continue to emit the legacy-compatible top-level columns that consumption
SDK 0.3.x already understands.
2. **Leave `attributes.adk` absent.** Stamping `attributes.adk.schema_version`
on rows that have no ADK provenance would falsely imply they came through
ADK and confuse consumers that key off the `attributes.adk.*` namespace.
Producer identity stays in `attributes.writer.agent` and
`attributes.source`. Revisit only if #190 explicitly defines a
cross-producer schema-version contract.
3. **Do not emit ADK 2.0-only event types in v0.1.** `AGENT_TRANSFER`,
`EVENT_COMPACTION`, and `AGENT_STATE_CHECKPOINT` are reserved for the ADK
producer path and stay out of scope for these wrappers.
4. Be exercised against the null-safe consumer tests tracked in #219 so
producer rows from this move do not regress ADK 2.0 consumer queries.

**Acceptance criterion:** the ADK 2.0 consumer views being added under
#212–#218 return correct results when `agent_events` contains rows from these
producers alongside ADK-produced rows.

## Migration milestones

1. **License / contribution check**
- Source repo is MIT-licensed; target repo is Apache-2.0 under Google LLC.
The current rights holder / original author of the moved files in
`Erroration2022/bigquery_agent_analytics_skill` opens the porting PR.
- Confirm CLA and license handling before copying files.
- Decide whether copied files keep attribution headers, are relicensed by
contribution, or are imported through a clean PR by the original author.

2. **Create producer package scaffolding**
- Add separate `bigquery-agent-analytics-tracing` packaging.
- Add base and extras dependencies.
- Add console scripts for `bqaa-codex`, `bqaa-drain`, and `bqaa-setup`.

3. **Port shared writer and drainer**
- Move `BQAAConfig`, logger, schema helper, spool/drain code, and state
store into package modules.
- Replace path assumptions like `sdk/python -> ../../scripts/bqaa_drain.py`
with package-resource or console-script based lookup.
- **Stop hardcoding `WRITER_PLUGIN_VERSION`.** Today it is a literal
`"0.1.0"` constant in `bqaa_tracing.py` that ends up in every row's
`attributes.writer.version`. Derive it from package metadata
(`importlib.metadata.version("bigquery-agent-analytics-tracing")`); when
the package is not installed (vendored plugin, local dev checkout), fall
back to `"0.0.0+local"` so adoption queries can cleanly filter out
non-release traffic. Preserve the existing `BQAA_WRITER_LABEL` env
override behavior. Propagate the same resolved version into the Claude
plugin artifact's `plugin.json` at build time so adoption queries against
`attributes.writer.version` reflect real releases.

4. **Port producers**
- OpenAI Agents processor.
- Codex `exec --json` wrapper.
- Claude hook adapter and hook script entry point.
- Setup script.

5. **Port plugin artifacts**
- Claude Code plugin manifest, hooks, commands, and setup skill.
- Codex plugin metadata as setup/command packaging only.
- Add an artifact build step for the Claude marketplace package.

6. **Tests**
- Convert script-style tests to pytest.
- Add unit tests for row shape, writer attribution, env config, spool file
envelopes, drainer grouping, Codex event mapping, and OpenAI span mapping.
- Keep live BigQuery smoke tests opt-in/gated by env.
- Add plugin packaging smoke test that verifies hooks can import their
bundled Python code without requiring global pip installation.
- **Vendored plugin drainer smoke** — prove `hook → spool → drainer →
BigQuery` works end-to-end when the tracing wheel is **not** installed
on `BQAA_PYTHON`, using either `PYTHONPATH` injection on the spawned
drainer environment or a vendored `bqaa_drain_entry.py` wrapper. This
is the failure mode `-m bigquery_agent_analytics_tracing.drain` alone
does not cover.

7. **CI and release**
- Add pytest matrix for base plus extras.
- Add formatting consistent with this repo (`pyink`, `isort`).
- Add Trusted Publishing for `bigquery-agent-analytics-tracing`.
- Cut `0.1.0`.

8. **Docs and cutover**
- Add `docs/producers/` or equivalent producer docs.
- Cross-link from README/SDK docs where users ask "how do I produce
`agent_events` rows?"
- Update the source repo README to point to this repo after release.
- Archive or freeze the migrated tracing tree in the source repo; keep the
query skill there if it is still maintained separately.

## Non-goals for the initial move

- No new top-level BigQuery columns in v0.1; any `attributes.*` contract
changes follow #190.
- No event-mapping behavior changes beyond packaging, import, and runtime
bootstrap changes (dynamic versioning, plugin artifact generation,
dependency preflight, drainer invocation rework are all in scope).
- No Codex TUI tracing unless Codex exposes a runtime trace surface.
- No claim that Claude Agent SDK has a separate adapter until one is actually
implemented.
- No merge into the existing `bigquery-agent-analytics` consumption wheel.

## Open questions for maintainers

1. Should the second PyPI distribution live under `producers/pyproject.toml`,
or should this repo use a multi-package Hatch configuration at the root?
2. Preferred top-level name: `producers/`, `integrations/`, or
`tracing/`? Recommendation: `producers/`, because this repo is already the
consumption SDK and these components produce `agent_events` rows.
3. Confirm the proposed license/CLA path with maintainers / Google legal — the
plan under Migration milestone #1 is for the current rights holder of the
moved files to open the porting PR after signing the Google CLA, with files
relicensed Apache-2.0 on contribution. Flag now if a different path is
required.
4. Should `storage-write` be an extra or included by default? Recommendation:
extra, because the code already supports a BigQuery client fallback and
`pyarrow` is a heavy dependency for local hook users.
5. Should Claude Agent SDK get a tiny helper API in v0.1, or stay documented as
loading the Claude Code plugin directly?
6. Marketplace order: release PyPI first, or ship Claude plugin first with
vendored Python? Recommendation: plugin can ship first if the artifact
vendors its Python code, but PyPI first reduces duplicated docs and import
churn.