GoogleCloudPlatform / GoogleCloudPlatform/BigQuery-Agent-Analytics-SDK

CLI: bqaa-materialize-window for scheduled graph refresh

Open
#161 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
47
Forks
21
Avg merge
2d 13h
Merged PRs (30d)
33

Description

## Problem

Customers running the SDK in production want to keep their MAKO graph fresh on a schedule, but today's `ontology-build` requires them to pass `--session-ids` explicitly. That forces the customer to write their own discovery query, schedule it, and pipe the results — work that should be the SDK's job.

The customer thinks in **time windows** ("materialize the last 6 hours"), not session IDs. We should offer that directly.

This issue is the operational follow-up to #107 (the four-guarantee notebook is now live end-to-end via #155 + #157 + #160; the next product slice is making the demo's pipeline ergonomic to schedule).

## Proposed CLI

```
bqaa-materialize-window \
--project-id my-project \
--dataset-id analytics \
--ontology ontology.yaml \
--binding binding.yaml \
--events-table agent_events \
--lookback-hours 6 \
--skip-property-graph \
--validate-binding \
--format json
```

### Flags (v0 + near-term)

| Flag | Purpose | Default |
|---|---|---|
| `--project-id` / `--dataset-id` | Standard BQ target | required |
| `--ontology` / `--binding` | YAML paths | required |
| `--events-table` | Source telemetry table | `agent_events` |
| `--lookback-hours` | Hours back from now | required (no default — explicit is safer) |
| `--state-table` | Checkpoint table (FQN). Writes `(last_checkpoint, run_id, sessions_materialized, ok)` rows. **Near-term, not optional later** — without it every run repeats the full lookback. | `_bqaa_materialization_state` (in `--dataset-id`) |
| `--overlap-minutes` | Re-process events newer than `(last_checkpoint - overlap_minutes)` to catch late-arriving rows | `15` |
| `--completion-event-type` | Treat sessions as done when this `event_type` appears. **Parameterized** so non-BQ-AA-plugin emitters aren't locked out. | `AGENT_COMPLETED` |
| `--include-active-sessions` | Materialize sessions without a completion event (partial coverage) | off |
| `--skip-property-graph` | Same semantic as `ontology-build` | **on by default** (refresh, not setup) |
| `--validate-binding` | Pre-flight binding-validate before extraction | on by default |
| `--bundles-root` | Compiled bundle directory | env `BQAA_BUNDLES_ROOT` or unset |
| `--reference-extractors-module` | Dotted module path for fallback | unset |
| `--dry-run` | Discover + validate; don't extract/materialize | off |
| `--format` | `json` / `text` | `json` |

## Behavior

### 1. Discover sessions

```sql
SELECT DISTINCT session_id
FROM `{project}.{dataset}.{events_table}`
WHERE timestamp >= GREATEST(
@last_checkpoint_minus_overlap,
TIMESTAMP_SUB(CURRENT_TIMESTAMP(), INTERVAL @lookback_hours HOUR)
)
AND timestamp < CURRENT_TIMESTAMP()
AND session_id IS NOT NULL
-- If --completion-event-type is set and --include-active-sessions is off:
AND EXISTS (
SELECT 1 FROM `{project}.{dataset}.{events_table}` e2
WHERE e2.session_id = e.session_id AND e2.event_type = @completion_event_type
)
```

**MUST leverage partition pruning** — the plugin partitions on `timestamp` (DAY); the `WHERE timestamp >=` filter prunes by day.

### 2. Extract full sessions

The window is for **session discovery**, not row filtering. A session that started before the window but completed inside it is materialized whole. Same `OntologyGraphManager.extract_graph(session_ids=...)` semantics as `ontology-build`.

### 3. Materialize

`OntologyMaterializer.materialize_with_status(graph, session_ids)`. Session-level delete-then-insert idempotency keeps re-runs safe.

### 4. Checkpoint

On success: write a row to `--state-table` with `(last_checkpoint=NOW, run_id, sessions_materialized=N, ok=true)`. Read it on next run to compute `last_checkpoint_minus_overlap`.

### 5. Partial-failure handling

If extraction fails on session N of M:
- Materialize sessions 1..N-1 (already extracted).
- Write a checkpoint at the timestamp of the last fully-materialized session.
- Exit non-zero with structured JSON error (which session, why).
- Next run picks up at that timestamp + overlap → at-least-once with idempotent retries.

## JSON report shape

```json
{
"run_id": "01HZ...",
"window_start": "2026-05-15T08:00:00Z",
"window_end": "2026-05-15T14:00:00Z",
"checkpoint_read": "2026-05-15T07:45:00Z",
"checkpoint_written": "2026-05-15T14:00:00Z",
"sessions_discovered": 42,
"sessions_materialized": 42,
"nodes_extracted": 123,
"edges_extracted": 98,
"rows_materialized": {"DecisionExecution": 42, "...": "..."},
"table_statuses": {"...": {"cleanup_status": "deleted", "insert_status": "inserted"}},
"compiled_bundle_fingerprint": "ab583...",
"compiled_outcomes": {
"compiled_unchanged": 168,
"compiled_changed": 0,
"fallback_fired": 0
},
"ok": true
}
```

`compiled_outcomes` is the rollout-health metric — operators watch `fallback_fired` to see when the compiled bundle stops covering production.

## Deployment recommendation

**Cloud Scheduler + Cloud Run Job** is the right operational story:

- Cloud Scheduler triggers a Cloud Run Job every N hours (config'd via cron).
- The job runs `bqaa-materialize-window --lookback-hours N --state-table ...`.
- The BigQuery property graph object stays fixed (user-owned per #104).
- The SDK only refreshes the underlying node/edge tables.
- The Cloud Run Job's exit code drives Cloud Scheduler's retry policy.

**Why not BigQuery Scheduled Queries?** Extraction/materialization is Python SDK orchestration. It may load compiled bundles, call `AI.GENERATE`, and write multiple tables. Scheduled Queries are SQL-only.

Ship a sample `cloudbuild.yaml` + `Dockerfile` + Cloud Run Job spec alongside the CLI so customers can `gcloud run jobs deploy` directly.

## Idempotency contract

The materializer already scopes cleanup by `session_ids`. Repeated runs over overlapping windows are safe by construction:
- Session in window N AND window N+1 → DELETE (by session_id) + INSERT idempotently in both runs.
- No raw time-window DELETEs — those would race with the streaming buffer and corrupt data.

This means the `--overlap-minutes` window is harmless from a correctness standpoint; it only costs the AI.GENERATE re-extraction tokens for the overlap period.

## Open questions

1. **State-table schema** — what columns do we want for run telemetry? Minimum: `run_id, window_start, window_end, last_checkpoint, sessions_materialized, ok, error_detail`. Should we also write per-table row counts for trend tracking?
2. **Permissions surface** — the CLI needs `bigquery.tables.update` on every materialized table + `bigquery.tables.create` on `--state-table` first run. Document the IAM minimum.
3. **Cost guardrails** — should `--dry-run` estimate the AI.GENERATE token cost before materializing? Could reuse Beat 3.7's per-session transcript-size estimate.
4. **Backfill mode** — a `bqaa-materialize-window --backfill --from 2026-05-01 --to 2026-05-15 --batch-hours 24` mode for first-run rollouts? Probably out-of-scope for v0; file separately.

## Acceptance criteria

- [ ] `bqaa-materialize-window --help` lists every flag in the table above.
- [ ] First-run (no state table) bootstraps the state table and materializes the full lookback window.
- [ ] Subsequent runs honor `last_checkpoint - overlap_minutes` as the lower bound.
- [ ] Sessions discovery query uses partition pruning (verified by `EXPLAIN`).
- [ ] Session-level delete-then-insert idempotency demonstrated by re-running the same window twice → identical materialized state.
- [ ] Partial failure halfway through materialization advances the checkpoint conservatively + exits non-zero.
- [ ] `--dry-run` discovers sessions + binding-validates but doesn't extract or materialize.
- [ ] JSON report includes `compiled_outcomes` when `--bundles-root` is set.
- [ ] Integration test against a live scratch dataset covers a 3-session lookback window twice (idempotency proof).
- [ ] Sample Cloud Run Job spec lands under `examples/` or `deploy/` so customers can copy-paste-deploy.

## Out of scope (this issue)

- Multi-tenant runs (one CLI invocation per `(project, dataset, ontology)` is fine for v0).
- Real-time / streaming materialization (this is a batch-window tool).
- Backfill mode (file as follow-up if needed).
- BQ Scheduled Query backend (the Python orchestration is the point).

## Related

- #107 — the four-guarantee notebook (storyboard now live end-to-end via #155 + #157 + #160).
- #104 — `--skip-property-graph` (the contract this CLI relies on).
- #75 — compiled extractors (the `--bundles-root` integration).
- #105 — binding-validate (the pre-flight before extraction).

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.