GoogleCloudPlatform / GoogleCloudPlatform/BigQuery-Agent-Analytics-SDK
Support a configurable custom_metadata key allowlist (only a2a:* keys captured today)
- Dominant language
- Python
- Stars
- 47
- Forks
- 21
- Avg merge
- 2d 13h
- Merged PRs (30d)
- 33
Description
## Problem
The ADK Python BQAA plugin only ever reads a fixed, hardcoded set of keys out of `event.custom_metadata`. Across the entire plugin, `event.custom_metadata` is consumed in exactly one place, and only `a2a:*`-prefixed keys survive:
```python
# third_party/py/google/adk/plugins/bigquery_agent_analytics_plugin.py
meta = getattr(event, "custom_metadata", None)
if meta and (
meta.get("a2a:request") is not None
or meta.get("a2a:response") is not None
):
a2a_keys = {k: v for k, v in meta.items() if k.startswith("a2a:")}
...
# logged into attributes.a2a_metadata
```
Two gates drop everything else:
1. The block only runs when `a2a:request` / `a2a:response` are present, so an event carrying *only* non-A2A metadata never enters it.
2. Even inside, the comprehension keeps only `k.startswith("a2a:")`.
As a result, any custom metadata an agent attaches to an event under a non-`a2a:` key is silently discarded and never reaches a queryable column or the `attributes` JSON.
## Motivating use case
UDR (Unified Deep Research) attaches resolved citations to its final report event via `event.custom_metadata['citation_metadata']` (the citation -> SQL/source mapping). Teams running evals locally with BQAA want this in their trace tables, but because `citation_metadata` is not an `a2a:` key, it is dropped before it can be logged. There is no mention of "citation" anywhere in the plugin today, and the final-response (`AGENT_RESPONSE`) path does not read `custom_metadata` at all.
This is not specific to citations — it is the general gap: there is no supported way for users to surface arbitrary `custom_metadata` into BQAA.
## Proposed direction
Add a config option to allowlist arbitrary `custom_metadata` keys, instead of hardcoding `a2a:`. For example:
```text
custom_metadata_allowlist: list[str] # exact keys and/or explicit prefix patterns
```
Behavior:
- For every event, capture the allowlisted keys from `custom_metadata` into `attributes.custom_metadata.*` (JSON), reusing the existing smart-truncation path (`_recursive_smart_truncate` + `max_content_length`).
- Keep the existing `a2a:*` handling as-is (or express it as a built-in default entry) for backward compatibility.
- Empty/unset allowlist preserves today's behavior exactly (no new data captured by default).
A user would then set e.g. `custom_metadata_allowlist=["citation_metadata"]` and query it via `JSON_QUERY(attributes, '$.custom_metadata."citation_metadata"')` — using a **quoted JSONPath segment**, mirroring how the A2A views already address keys with punctuation (`$.a2a_metadata."a2a:request"`).
**Refinements (validated against ADK `main` @`c007a874`):**
1. **Explicit prefix syntax, not implicit.** Make prefix matching a declared form (e.g. `"a2a:*"` or `"prefix:a2a:"`) rather than treating any allowlist string as a possible prefix — so `"citation_metadata"` only ever matches the exact key.
2. **Reuse the plugin's full safety pipeline on user-allowlisted values**, not just truncation. A user can allowlist a key holding secrets, so captured metadata must go through: `_recursive_smart_truncate(..., max_content_length)`, sensitive-key **redaction** (`_SENSITIVE_KEYS` = `client_secret` / `access_token` / `refresh_token` / `id_token` / `api_key` / `password`, plus `temp:`-prefixed keys), circular-reference handling (`[CIRCULAR_REFERENCE]`), `is_truncated` propagation, and the `json.dumps(..., default=str)` fallback.
3. **Define *which rows* receive the metadata.** Today `custom_metadata` is read in exactly one place (`on_event_callback`, gated on `a2a:request`/`a2a:response`), and the final-response `AGENT_RESPONSE` path does **not** read it at all — which is exactly the UDR motivating case. The plan should copy allowlisted keys into every BQAA row emitted from that source `Event` (notably `AGENT_RESPONSE`, plus `STATE_DELTA`, A2A/HITL, etc.). If instead a dedicated `CUSTOM_METADATA` row is emitted, call that out explicitly — it changes query ergonomics.
4. **Preserve A2A as a built-in path, not just a default allowlist entry.** The `A2A_INTERACTION` semantic event type, content selection from `a2a:response`, and the typed view columns (`a2a_task_id`, `a2a_context_id`, `a2a_request`, `a2a_response`) must stay intact. Generic capture is additive, under the separate `attributes.custom_metadata.*` namespace; `attributes.a2a_metadata` is unchanged.
5. **Don't hardcode citation-specific behavior.** Citations are the motivating use case but the first pass should stay a generic allowlist; promotion of hot keys to typed columns is a later, separate step.
## Acceptance criteria
- New config field that captures listed `custom_metadata` keys (exact + explicit prefix patterns) into `attributes.custom_metadata.*`.
- Default behavior unchanged when the allowlist is empty/unset — byte-for-byte, including that non-`a2a:` keys (e.g. `citation_metadata`) remain dropped.
- Existing `a2a:*` capture, the `A2A_INTERACTION` event type, and its typed view columns are preserved; generic capture lives under a separate namespace.
- Captured values pass the full safety pipeline: truncation (`max_content_length` + `is_truncated`), sensitive-key redaction, and circular-reference handling.
- The plan states explicitly which rows carry the metadata (incl. `AGENT_RESPONSE`, which does not read `custom_metadata` today).
- Broaden the `is_truncated` schema/column description from "the `content` field was truncated" to cover **content *or* metadata payload truncation** — allowlisted-metadata truncation flips the same flag (A2A metadata already uses this pattern).
- Document the query pattern with **quoted JSONPath segments** (`JSON_QUERY(attributes, '$.custom_metadata.""')`), since allowlisted keys may contain `:` / `.` / other punctuation that breaks an unquoted `$.custom_metadata.` path.
- Negative tests:
- no allowlist → current behavior exactly (no `citation_metadata`);
- allowlisted `citation_metadata` appears under `attributes.custom_metadata.citation_metadata`;
- non-allowlisted metadata is absent;
- `a2a:*` behavior and existing typed view fields are unchanged;
- long metadata flips `is_truncated`; sensitive metadata is **redacted** (`[REDACTED]`) without flipping the flag; circular / deeply-nested metadata is handled safely (`[CIRCULAR_REFERENCE]`) — `is_truncated` is set **only when string-length truncation actually occurs** (matching `_recursive_smart_truncate`'s current semantics, which flip the flag on truncation but not on redaction or circular-ref handling).
## Notes
Optionally, frequently-used keys (e.g. citations) could later be promoted from `attributes.custom_metadata.*` to typed SDK/view columns once the shape proves useful, the same staged approach suggested in #312.
Contributor guide
Research direction
Start in third_party/py/google/adk/plugins/bigquery_agent_analytics_plugin.py at on_event_callback, tracing the AGENT_RESPONSE and other Event row paths alongside the existing A2A handling and safety helpers. Verify the configuration, schema description, documentation, and plugin tests cover exact and explicit-prefix allowlists, safe metadata processing, quoted JSONPath queries, unchanged defaults, and metadata on every specified row.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- observability
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Clearly specified
- Newbie friendliness
- 68/100