GoogleCloudPlatform / GoogleCloudPlatform/BigQuery-Agent-Analytics-SDK

Support column-level filtering / projection of the BQAA schema

Open
#321 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
47
Forks
21
Avg merge
2d 13h
Merged PRs (30d)
33

Description

## Problem

The BQAA producer schema is all-or-nothing: when a plugin is enabled, every event type and every populated column is written to the customer's BigQuery table. Users have no supported way to restrict which columns (or which parts of the payload) get logged.

This matters for several reasons:

- **Sensitivity / governance:** some teams want telemetry (latency, tool calls, errors, hierarchy) but must avoid persisting heavy content columns (e.g. `content`, large `attributes` payloads) for PII/compliance reasons.
- **Cost & noise:** content and attributes JSON can dominate row size; teams that only need structural/timing signals pay for columns they never query.
- **Stability:** consumers building views/dashboards benefit from declaring the exact column projection they depend on.

Today the only knobs are coarse (e.g. content-capture on/off, `max_content_length`); there is no per-column selection.

## Proposed direction

Add column-level filtering/projection config for the BQAA schema, e.g.:

```text
column_allowlist: list[str] # only write these columns
# and/or
column_denylist: list[str] # write everything except these
```

Behavior:

- Applied at write time, so excluded columns are never sent to BigQuery (not just hidden in a view).
- Always-required correlation/identity columns (e.g. `trace_id`, `span_id`, `parent_span_id`, `event_type`, timestamps) are protected and cannot be filtered out, to keep rows joinable and well-formed.
- Mutually exclusive allowlist/denylist with clear precedence; unset = today's full schema (no behavior change).
- Plays well with existing content-capture / truncation options.

**Refinements (validated against ADK `main` @`c007a874`).** The current plugin assumes a fixed physical schema in several places — `_get_events_schema()`, `to_arrow_schema(self._schema)` (Storage Write API), the full row dict built in `_log_event()`, the per-event views, and the additive-only `_maybe_upgrade_schema()` — so projection must be designed conservatively:

1. **Denylist-first for v1.** A full allowlist is easy to mis-specify into broken rows/views; most governance/cost wins are "don't persist `content` / `content_parts` / large `attributes`". Scope v1 to a narrow, typed denylist of payload columns and defer a general allowlist:
```python
payload_column_denylist: list[Literal["content", "content_parts", "attributes", "latency_ms"]]
```
2. **Protected column set that cannot be filtered.** All 12 of these exist in `_get_events_schema()` and are exactly the `_VIEW_COMMON_COLUMNS` the views depend on: `timestamp`, `event_type`, `agent`, `session_id`, `invocation_id`, `user_id`, `trace_id`, `span_id`, `parent_span_id`, `status`, `error_message`, `is_truncated`. Denylisting any of them raises a clear `ValueError`.
3. **View degradation is the hard case — and it is not only `attributes`.** The current view definitions read JSON paths from **`content`** (~34 expressions), **`attributes`** (~25, including `attributes.adk.*` and `attributes.a2a_metadata.*`), **and `latency_ms`** (~6, e.g. `CAST(JSON_VALUE(latency_ms, '$.total_ms') AS INT64)`). General rule: **any denied column referenced by a view expression must either drop the dependent derived view column(s) or disable/reject view creation with a clear warning** — projection has to be view-aware for `content`, `attributes`, and `latency_ms` alike, not just `attributes`. Silent broken views are worse than no views. (`create_views=False` sidesteps this entirely.)
4. **Schema-first, not row-only.** Apply the projection before **table creation**, **Arrow schema** construction, **row serialization**, the **auto-schema-upgrade diff**, and **view generation** — so the table schema, the Arrow schema, and the row dict never disagree (a full-schema table with key-omitting rows, or vice versa, makes the Write API and view SQL brittle).
5. **Keep event-type filtering separate.** The existing `event_allowlist` / `event_denylist` already handle *which event types* are written; this issue is *physical column projection* only and should reference event filtering as prior art rather than fold both into one config.

## Open questions (resolved direction)

- **Allowlist vs denylist vs both** → **denylist-first** for v1 (scoped `payload_column_denylist`); a general allowlist is deferred.
- **Also exclude entire event types?** → **No, keep separate.** `event_allowlist` / `event_denylist` already cover event-type filtering; this issue is column projection only.
- **View interaction** → views must **degrade gracefully or be disabled** with a clear warning; any view-referenced column (`content` / `attributes` / `latency_ms`) may only be dropped when views are off or projection-aware (see Refinement 3).

## Acceptance criteria

- Config to restrict which payload columns are persisted (denylist-first, scoped to `content` / `content_parts` / `attributes` / `latency_ms` in v1).
- Protected set of 12 join/identity columns that cannot be removed; denylisting one raises a clear `ValueError`.
- Projection is applied **schema-first** — table schema, Arrow schema, row dict, auto-schema-upgrade diff, and views all stay consistent.
- Unset config preserves today's full schema **byte-for-byte** (same row keys + schema).
- Dropping any view-referenced column (`content`, `attributes`, `latency_ms`) is only permitted with `create_views=False` or projection-aware views; otherwise rejected.
- Documented precedence rules and query/view implications.
- Broaden the `is_truncated` schema/column description so it no longer ties the flag to the `content` field specifically — `content` may be denied via projection, and (with the companion `custom_metadata` allowlist) metadata payloads can also be truncated. Wording such as "content or metadata payload was truncated".
- Tests for failure modes:
- denylisting a protected column raises `ValueError`;
- denylisting `content` removes it from table schema, Arrow schema, row dict, and views;
- default config produces the same row keys/schema as today;
- existing full-schema tables keep working when projection is unset;
- projection + `auto_schema_upgrade` never attempts to drop columns from existing tables;
- view-degradation tests for `content`, `attributes`, **and** `latency_ms` individually: denying each either drops its dependent derived view columns or disables view creation with a clear warning.

## Notes

Filed alongside the `custom_metadata` allowlist request (see companion issue): one controls which *extra* keys get *added* into `attributes`, this one controls which *existing* columns get *written*. Together they let users tune exactly what BQAA persists.

Contributor guide

Open the contributing guide

Research direction

Start by tracing _get_events_schema(), to_arrow_schema(self._schema), _log_event(), _maybe_upgrade_schema(), and the existing per-event view definitions. Use event_allowlist/event_denylist as prior art, then verify that projection keeps the table schema, Arrow schema, row dict, upgrades, and view behavior consistent for content, attributes, and latency_ms; run the specified failure-mode and default-schema tests when implemented.

Written by the indexing model from the issue text.

Assessment

Tech stack
google-cloud, python
Domain
backend, cloud, databases
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.