BQAA Java preview readiness tracker: correctness, lifecycle, and Python contract parity

## Summary

This is the consolidated readiness tracker for the Java BigQuery Agent Analytics (BQAA) plugin.

Current verdict: **not ready to exit Preview or claim Python-compatible Preview behavior**. Full Python feature parity is not required, but the P1 items below can produce incorrect analytics, duplicate telemetry, retained per-invocation resources, or incomplete cleanup on supported Java runtime paths.

Latest comparison, pinned on 2026-07-10:

- Java `google/adk-java@08f4fdc6c9ea80ec46723b2c6b71a4409c77feed`
- Python `google/adk-python@9d306f5d322eccbc5022ac72407fa8a994f64cf6`

## P1: must fix before broad/public Preview

- [ ] **Make trace state safe for concurrent `ParallelAgent` branches.**
- Java stores one `TraceManager` per invocation and one global LIFO stack inside it.
- `ParallelAgent` merges concurrently scheduled branches with the same invocation ID. If branch A starts, branch B starts, and A completes first, A pops B's span.
- Use task-local, branch-keyed, or otherwise ownership-checked trace state.
- Add a deterministic same-invocation overlap test that completes branches out of start order and validates span IDs, parents, and STARTING/COMPLETED pairing.

- [ ] **Stop creating plugin-owned OpenTelemetry spans.**
- `TraceManager.pushSpan()` starts a real OTel span and pop/cleanup ends it.
- With an SDK exporter configured, BQAA therefore exports a duplicate analytics span tree alongside ADK's framework spans.
- Match Python's ID-only internal execution tree while inheriting an ambient `trace_id` when available.
- Add an in-memory exporter regression test asserting BQAA exports zero plugin-owned spans.

- [ ] **Apply sensitive-key redaction at the final attributes boundary.**
- Content and session-metadata paths now use the redacting formatter, but the final assembled attributes map is not passed through that boundary.
- `state_delta`, custom tags, labels, and other extra attributes can therefore carry sensitive-key values into BigQuery unchanged.
- Redact the complete attributes tree immediately before serialization, regardless of which producer populated it. Python currently has the same gap, so parity is not a safety defense.
- Add nested secret-key fixtures for state deltas, custom tags, labels, and generic metadata.

- [ ] **Fix per-invocation processor/task lifecycle.**
- `BatchProcessor.start()` discards the `ScheduledFuture` returned by `scheduleWithFixedDelay()`.
- Per-invocation `close()` does not cancel it, so every completed invocation leaves a periodic task retaining its closed processor/writer until plugin-wide shutdown.
- Store/cancel the task during idempotent close and add a high-cardinality invocation test proving scheduled work/resources are released.

- [ ] **Make append, final drain, and timeout cleanup genuinely bounded.**
- `writer.append(...).get()` has no total deadline, and final processor drain runs outside the effective `shutdownTimeout` bound.
- Pending parse/offload futures are not cancelled or lifecycle-gated after timeout. A late completion can recreate a `BatchProcessor` for an invocation that has already been finalized, with no later invocation cleanup.
- Apply a total append/drain deadline, reject/account appends after invocation close, and test a never-completing append plus a late GCS/parse completion.

- [ ] **Surface and account writer-construction and continuation failures.**
- `StreamWriter` creation occurs inside `computeIfAbsent`; if the future completes exceptionally, the pending task is removed without a durable error signal or drop counter.
- Rows can therefore disappear before a processor exists, outside the current loss-accounting path.
- Record a specific drop reason, emit an observable error, and make late/exceptional continuations respect the invocation lifecycle gate.

- [ ] **Correct HITL and long-running-tool pause/resume semantics.**
- Java currently labels both HITL `FunctionCall` requests and `FunctionResponse` results as `*_COMPLETED`.
- User-message handling inspects `FunctionCall` where resumed input arrives as `FunctionResponse`.
- Emit pairable `TOOL_PAUSED` / resumed `TOOL_COMPLETED` rows with `function_call_id` and `pause_kind`, without double-routing HITL completions.
- Add a Runner-level two-turn test for HITL and non-HITL long-running tools.

- [ ] **Add terminal agent/run failure analytics and guaranteed cleanup.**
- Java's plugin API exposes model/tool error callbacks but no agent/run error callbacks.
- `afterAgent` / `afterRun` are concatenated after successful completion, so an upstream exception can skip `AGENT_ERROR`, `INVOCATION_ERROR`, per-invocation flush, and trace cleanup.
- Add framework error hooks and finally-style invocation cleanup that preserves the original exception.

## Contract gaps: fix or explicitly exclude from a narrow Preview

- [ ] **Align or version the `AGENT_RESPONSE` row/view contract.**
- Java emits `content.text_summary` and exposes `text_summary` in its view.
- Python emits `content.response` and exposes `response_text`.
- The same event type should be queryable across SDKs; add a cross-language golden-row test.

- [ ] **Complete supported workflow taxonomy.**
- Java runtime models transfer, compaction, and long-running tool IDs, but Java BQAA does not emit Python-equivalent `AGENT_TRANSFER`, `EVENT_COMPACTION`, and pairable `TOOL_PAUSED` rows/views.
- Implement these before a parity claim, or list each omission in Preview scope documentation.

- [ ] **Add the minimum `attributes.adk` producer envelope.**
- At minimum stamp schema version, app name, and available source-event identity/branch fields.
- Python-only node/scope fields can follow when Java exposes equivalent runtime data.

- [ ] **Publish an executable Java Preview guide and data contract.**
- Cover dependency/setup, ADC and IAM, dataset/table/view defaults, explicit close behavior, drop-stat monitoring, GCS/privacy controls, event taxonomy, row/view contract, and documented Python-parity exclusions.

## P2 reliability and operational readiness

- [ ] Move first-use table bootstrap off the agent callback thread; coalesce concurrent setup and use bounded retry/backoff after failures.
- [ ] Make `enabled=false` avoid ADC resolution and BigQuery client initialization.
- [ ] Add explicit loss accounting for setup-not-ready, writer construction, parse timeout, and rejected late continuations.
- [ ] Add a gated real-BigQuery smoke test using public construction, real Storage Write append/drain, and query readback. The current `BigQueryAgentAnalyticsPluginE2ETest` mocks BigQuery, `BigQueryWriteClient`, and `StreamWriter`.
- [ ] Decide and document Java/Python default differences (`agent_analytics.events`, views disabled vs Python `agent_events`, views enabled).
- [ ] Make model, tool, and BigQuery exception-message capture null-safe; Java exceptions may legally return a null message, while the generated event value objects require a non-null optional payload.
- [ ] Validate all duration and retry settings at config build time, including positive flush/shutdown durations and valid retry-attempt bounds.

## Lower-priority Python gaps that do not independently block a narrow Preview

- [ ] Schema-first payload-column projection / denylist.
- [ ] Generic custom-metadata allowlisting.
- [ ] Full node/scope/route/render/rewind envelope parity.
- [ ] Structured LLM tool declaration parity.
- [ ] Public flush/view-management helper parity.

These should remain roadmap items unless the Preview explicitly promises full Python query/config parity.

## Confirmed fixed on current Java `main`

- [x] Sensitive-key redaction for content/session-metadata paths and fail-closed custom content formatting.
- Final assembled attributes remain open under the P1 above and are not included in this completed item.
- [x] Empty state deltas no longer emit `STATE_DELTA`.
- [x] Timestamp partitioning for newly created tables.
- [x] Failed table ensure is retryable; schema drift is logged.
- [x] Root agent initialization and `Event.author` fallback.
- [x] Drop-stat folding, regional routing, shutdown hook, headers, and numeric/config validation.

## Acceptance tests for Preview readiness

- [ ] Same-invocation `ParallelAgent` overlap preserves branch-local span ownership.
- [ ] Recording OTel exporter receives zero BQAA-owned spans.
- [ ] Final attributes redaction covers nested secrets in state deltas, custom tags, labels, and generic metadata.
- [ ] Many completed invocations leave no periodic flush tasks/processors/writers retained.
- [ ] `shutdownTimeout` bounds a never-completing append and final drain.
- [ ] A parse/offload future completing after timeout cannot recreate an invocation processor.
- [ ] Writer-construction failures increment a specific drop counter and emit an observable error.
- [ ] HITL and ordinary long-running tools produce correctly named and pairable pause/resume rows.
- [ ] Agent and invocation exceptions produce terminal rows, preserve the original error, and finalize state exactly once.
- [ ] Null-message model/tool/BigQuery exceptions do not break analytics callbacks.
- [ ] Invalid flush/shutdown durations and retry settings fail at config construction.
- [ ] Java/Python golden fixtures agree on common event names, row fields, content JSON, trace linkage, and view projections.
- [ ] Slow/failing first-use table setup does not stall agent progress or trigger one control-plane RPC per event.
- [ ] Real BigQuery write/readback smoke test passes in a disposable dataset.

## Current verification signal

- Java focused BQAA suite: **135 passed, 0 failed**.
- Java full core main profile: **1,591 passed, 0 failed/errors, 24 skipped**; additional profiles also passed.
- Python pinned plugin suite: **306 passed, 6 skipped**.
- These suites do not currently cover the production shapes listed in the acceptance checklist above.

## Pinned references

- Java plugin package: https://github.com/google/adk-java/tree/08f4fdc6c9ea80ec46723b2c6b71a4409c77feed/core/src/main/java/com/google/adk/plugins/agentanalytics
- Java `ParallelAgent`: https://github.com/google/adk-java/blob/08f4fdc6c9ea80ec46723b2c6b71a4409c77feed/core/src/main/java/com/google/adk/agents/ParallelAgent.java#L172-L184
- Python reference plugin: https://github.com/google/adk-python/blob/9d306f5d322eccbc5022ac72407fa8a994f64cf6/src/google/adk/plugins/bigquery_agent_analytics_plugin.py

Historical investigation and withdrawn/down-scoped findings remain available in the comments below.