Design: OpenTelemetry Collector BigQuery exporter — generic OTLP traces/logs → BigQuery (PRD & CUJs)

# Design Doc: `bigqueryexporter` for the OpenTelemetry Collector — generic OTLP → BigQuery (PRD + CUJs)

**Status:** v2.9 — resolves the [fresh solo v2.8 review](https://github.com/GoogleCloudPlatform/BigQuery-Agent-Analytics-SDK/issues/364#issuecomment-5040431393): mid-run destination-state failures get a bounded repair window; invalid UTF-8 has an explicit reject-before-hash policy; span dedup SQL and end-column modes are frozen; log namespaces have a deployable grammar and rollout contract; component telemetry has exact ownership/names; attempt timeout and queue sizing are tied to explicit batching. **No design P0 remains. The body is ready for the first §6.1 provenance sign-off, but M1 internal implementation may begin only after that sign-off is actually recorded (entry criterion, not "engagement started"). External repository creation/publication remains gated on GCP OSS release approval (§6). The M2 external Alpha publishes the schema as `v0alpha1`; v1 freezes only after design-partner validation (§7.5). Terminology: "initial release family" = `v0alpha1` through the v1 freeze; "v1" alone refers to the frozen compatibility contract.** Prior history: [design review](https://github.com/GoogleCloudPlatform/BigQuery-Agent-Analytics-SDK/issues/364#issuecomment-4978066434), [Q2/Q4](https://github.com/GoogleCloudPlatform/BigQuery-Agent-Analytics-SDK/issues/364#issuecomment-4978169336), [contract gaps](https://github.com/GoogleCloudPlatform/BigQuery-Agent-Analytics-SDK/issues/364#issuecomment-4978223545), [v2.3 review](https://github.com/GoogleCloudPlatform/BigQuery-Agent-Analytics-SDK/issues/364#issuecomment-5023275362), [v2.6 review](https://github.com/GoogleCloudPlatform/BigQuery-Agent-Analytics-SDK/issues/364#issuecomment-5039240293)
**Author:** BigQuery Agent Analytics team
**Upstream target (eventual):** [open-telemetry/opentelemetry-collector-contrib — exporter/](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/exporter)
**Prior art:** [contrib#34809](https://github.com/open-telemetry/opentelemetry-collector-contrib/issues/34809) (closed after inactivity) → implementation [contrib PR #46280](https://github.com/open-telemetry/opentelemetry-collector-contrib/pull/46280) (closed; stalled at the sponsor gate; continued externally by its author — not linked here per the §6.1 provenance protocol; repository metadata, if needed, lives in the controlled inspection ledger)

---

## 1. Product statement

> Export OTLP **traces and logs** to **versioned, documented BigQuery relational tables with an explicitly declared maturity level** (§7.5) through the **Storage Write API**, with Collector-native queue/retry behavior, predictable **at-least-once** semantics, safe table management, and production-grade self-observability.

Explicitly out of scope for this proposal: any BigQuery Agent Analytics (`agent_events`) transformation, schema modes, or GenAI semantic-convention mapping. If a BQAA transformation becomes valuable later, it will be designed separately (as a BQAA-owned processor/connector/exporter) in a follow-up issue. This exporter has one job: reliable OTLP-to-BigQuery transport with a versioned relational schema at a declared maturity level.

## 2. Background & problem statement

The OTel Collector has no BigQuery exporter. Current routes to get telemetry into BigQuery for SQL analytics:

- `googlecloudexporter` → Cloud Trace/Logging → Log Router sink → BigQuery: log-oriented schema not designed for span analytics, adds an intermediate product hop with its own cost and latency characteristics (to be quantified in the comparison doc rather than asserted here).
- Pub/Sub exporter + Dataflow/BigQuery subscription: works, but adds two managed services and a pipeline to operate.
- File export + `bq load`: batch-only, operationally manual.

Community demand exists: contrib#34809 requested the component; PR #46280 delivered a working implementation (traces, metrics, logs, Storage Write API, ingestion-time partitioned tables, unit + integration tests) but closed without landing because no sponsor materialized. The implementation is used in production by its author's organization and is maintained externally.

**Why this team owns it:** the BigQuery-Agent-Analytics-SDK is a consumption-layer toolkit over telemetry that lives in BigQuery — its reach is bounded by how easily telemetry gets *into* BigQuery. A first-class, generic OTLP ingestion path grows the data footprint every SDK capability (analysis, evaluation, curation, context graph) operates on, without coupling the exporter to any BQAA schema. This team already owns the BigQuery-telemetry ingestion/analysis boundary, holds the BigQuery domain expertise the exporter's correctness contracts require, and commits funded, named support ownership (the M2 support package) rather than best-effort maintenance. This rationale accompanies the OSS release request.

## 3. Goals

- G1: Export OTLP **traces** and **logs** to BigQuery relational tables with a versioned, field-level schema contract (§7).
- G2: Ingestion via the Storage Write API **default stream**, with an explicit at-least-once delivery contract and documented deduplication guidance (§8).
- G3: Standard collector ergonomics: `exporterhelper` sending queue (with optional persistent storage), retry, timeout; ADC / service-account / impersonation auth; self-observability via `metadata.yaml`-defined telemetry.
- G4: Optional, clearly-separated control-plane table management (`auto_create`) with fail-fast schema validation at startup.
- G5: External-first delivery: versioned Go module + custom OCB distribution + OTel registry listing, building the adoption evidence contrib's donation process expects.

## 4. Non-Goals

- N1: Metrics signal — future, independent proposal (the relational shape for metrics needs its own design; not a prerequisite for trace/log maturity).
- N2: Exactly-once semantics — future, independent proposal (committed/pending streams with offsets).
- N3: Any BQAA / GenAI / `agent_events` transformation (see §1).
- N4: Replacing `googlecloudexporter` — Cloud Trace/Monitoring/Logging remain the operational backends; BigQuery is the analytical backend.
- N5: Legacy `insertAll` streaming inserts. Note: `insertAll` is a legacy alternative, **not deprecated**; the choice of Storage Write API is based on its delivery semantics, throughput, and cost properties, compared explicitly in §12.

## 5. PRD

### 5.1 Personas

| Persona | Description |
|---|---|
| **P1 — Data engineer / analyst** (primary) | Wants raw traces/logs in BigQuery to join with business data using plain SQL / dbt / Looker, without operating a streaming pipeline. |
| **P2 — SRE / observability engineer** | Owns collector fleets. Cares about delivery guarantees, duplicate behavior, backpressure, quotas, cost, security of queued data, and self-monitoring. |
| **P3 — Platform / security engineer** | Governs what leaves the pipeline: redaction, residency, IAM, retention, audit. |

### 5.2 Functional requirements

| ID | Requirement | Priority |
|---|---|---|
| FR1 | Export OTLP traces to a configurable BigQuery table with the §7 span schema. | P0 |
| FR2 | Export OTLP logs to a configurable BigQuery table with the §7 log schema. | P0 |
| FR3 | Ingest via Storage Write API default stream through `managedwriter`, with bounded asynchronous in-flight appends (window size, memory budget, completion handling, shutdown drain, and stream-recreation behavior all specified — see §8). | P0 |
| FR4 | Config: `project`, `dataset`, `location`, `traces.table`, `logs.table`, credentials (ADC / credentials file / impersonation), endpoint override (emulator/dev only, with verified TLS and explicit insecure/no-auth flags). No mode enum — one schema, one exporter. | P0 |
| FR5 | `exporterhelper` integration: `sending_queue` (optional persistent via `file_storage`), `retry_on_failure`, and per-attempt backend `timeout`; exporter shutdown uses the separate `write.shutdown_drain_timeout` contract in §8.1. Exactly one retry owner per error class (exporterhelper vs managedwriter) prevents multiplied attempts and duplicate amplification. | P0 |
| FR6 | Row-error recovery, executable and aligned with actual Collector semantics (`consumererror.NewTraces/NewLogs` represent **retryable** subsets; exporterhelper drops permanent errors before partial-error handling): when `AppendRows` reports row errors, **no rows in that request are appended**. The exporter (a) removes, counts, and diagnoses permanently invalid rows **internally** (no DLQ in the initial release family — §8.5); (b) appends the valid subset; (c) returns `nil` on success; (d) returns **only retryable, still-unacknowledged telemetry** via [`consumererror.NewTraces`/`NewLogs`](https://pkg.go.dev/go.opentelemetry.io/collector/consumer/consumererror). Permanent rows are never routed back through `consumererror`. **Durability scope:** the narrowed retryable subset exists only inside one uninterrupted in-memory exporterhelper retry loop — the persistent queue stores the **original Collector request**, not a durably checkpointed residual. After a crash/shutdown during partial delivery, the original request may replay: permanently invalid rows are **deterministically re-filtered** (same rows rejected again, idempotently), and previously acknowledged valid rows **may duplicate** (covered by the §8.4 at-least-once contract and §7.4 dedup). "Acknowledged rows are never requeued" is therefore scoped to the uninterrupted in-memory loop. Collector release is **pinned**; M2 tests this as **two separate cases**: (i) in-process subset retry, (ii) crash replay. Durable residual checkpointing is a future custom-queue feature, explicitly not v1. | P0 |
| FR7 | Request sizing: **row-byte** checks are exact, post-protobuf-serialization. The **request-byte** bound is deliberately *not* described as exact — the public `managedwriter.AppendRows` API constructs the final request (schema + envelope overhead) internally, so the exporter cannot observe it. Contract: `max_request_bytes` bounds the **sum of serialized row payloads** with the 8 MiB headroom absorbing framing/schema overhead below the <10 MB API limit; an unexpected server oversize rejection is handled by **splitting and retrying**, not failing the batch. | P0 |
| FR8 | Structural pre-serialization validation: nesting depth, attribute count, event/link count, collection sizes, individual string/byte value sizes, and recursive valid-UTF-8 checks for every string and map key (encoded-size enforcement belongs to FR7 post-serialization). The exporter **rejects rather than repairs** invalid UTF-8, reason `invalid_utf8`; rejected bytes never enter `fp1` canonicalization or the row serializer. Violations permanently reject/drop only the offending record, with metadata-only diagnostics. | P0 |
| FR9 | Error matrix (see §8.3) covering serialization errors, row errors, schema mismatch, invalid stream/schema, quota exhaustion, transient service errors, permission failures, credential refresh, uncertain acknowledgement, and shutdown timeout — each with a classification (permanent/retryable), a retry owner, and an observable outcome. | P0 |
| FR10 | At-least-once contract surfaced in docs and config comments: default-stream delivery **may duplicate data**. Ship deterministic row identities and canonical deduplication SQL per §7.4: (`trace_id`,`span_id`) for spans; logs use a three-expression key `(has_exact_identity, namespace_or_empty, source_id_or_fingerprint)` so exact source identity and content fallback cannot collide. The namespace comes from deployment-controlled configuration, never from an arbitrary telemetry attribute; when absent, fingerprint fallback is table-wide. | P0 |
| FR11a | **Startup validation** (M1, required for CUJ-1's fail-fast promise): compatibility-label/epoch check per §7.5 (including missing/unparseable labels), schema compatibility, dataset-location and partition/clustering mismatch detection — confirmed incompatibility fails startup. Incompatible-table tests are M1/M2 exit criteria. | P0 |
| FR11b | Optional `auto_create` (M3): dataset and/or table creation as a separate control-plane concern (§9), with concurrent-create race handling and **explicit CMEK + partition-expiration configuration with mismatch validation — never silently inheriting project defaults**. | P1 |
| FR12a | Delivery-critical self-telemetry (required by the M2 reliability CUJ) is the exact inherited-versus-component metric contract in §8.6. Component metrics are declared in `metadata.yaml` with exact emitted names, types, units, and bounded labels; generated documentation plus an M1 metadata test freezes them. | P0 |
| FR12b | Extended operational telemetry (M3): sent bytes (counter), append-result latency (histogram), and remaining operational metrics, same `metadata.yaml` discipline. | P1 |
| FR13 | No raw telemetry in exporter logs — ever, including stable cross-system correlators: raw trace IDs, span IDs, source record IDs, and `record_fingerprint` values are **prohibited** in normal logs. Diagnostics carry destination, error code, row index, serialized size, and a **random append-operation ID** (or short-lived keyed HMAC) for correlation. A diagnostic field path is built only from allowlisted structural segments and numeric indexes (for example, `span.events[3].timestamp`); arbitrary resource/scope/record attribute keys or `kvlist` keys are never copied into logs — they are omitted or represented by a constant `` segment. | P0 |
| FR14 | Multi-dataset/tenant routing via `routingconnector` (documented recipe), keyed only on authenticated or deployment-controlled identity — never an application-controlled resource attribute used as authorization. | P1 |

### 5.3 Non-functional requirements

Performance NFRs are falsifiable: each ships with a reproducible workload definition (collector CPU/memory and region, dataset location, trace/log mix, average and p95 serialized row size, attribute/event/link cardinality, batch sizing, in-flight request count, test duration, allowed error rate).

| ID | Requirement |
|---|---|
| NFR1 | Throughput target: ≥ 10k rows/s per collector instance under the published reference workload (4 vCPU / 8 GiB, same-region dataset, mixed 70/30 span/log, p95 row ≤ 4 KiB); exporter must not be the bottleneck relative to Storage Write API quota. |
| NFR2 | p99 added export latency ≤ 5s under the reference workload, measured receiver-ingest → append acknowledgement, scoped to a healthy backend with no pre-existing queue backlog. |
| NFR3 | No loss of valid rows under transient BigQuery unavailability — with the no-loss horizon **explicitly bounded by both queue capacity and retry expiry** (`retry_on_failure.max_elapsed_time`; exporterhelper returns a terminal error at horizon expiry and the persistent queue then deletes the item). The reference config sets the horizon **above** the CUJ-2 outage duration (900s vs 600s). The outage criterion is falsifiable, not tautological: the published reference workload states batch size, measured persistent-queue bytes per batch, provisioned volume size and headroom, and requires **zero loss of valid rows for the complete 600-second reference outage** (≈6M logical rows at the 10k rows/s reference rate) — "up to queue capacity" alone would pass at any capacity. Bounded duplicate behavior on recovery per §8.4. Ships with an explicit **outcome matrix**: queue full (block vs drop policy stated), disk full, corruption, retry expiry, shutdown, crash replay. `sending_queue.sizer: requests` is explicit in the reference config, so `queue_size` is a request/batch-count bound rather than a row/byte capacity guarantee. With the explicit 1s batch timeout, `send_batch_size = send_batch_max_size = 8192`, no metadata partitioning, and the 70/30 reference mix, steady state produces about one trace plus one log Collector request per second: ≈1,200 queued requests over 600s, before headroom, versus `queue_size: 5000`. This arithmetic is a planning bound, not proof: M2 publishes measured request count and on-disk bytes and demonstrates the full outage. Storage-Write splits do not consume additional exporterhelper queue slots. |
| NFR4 | IAM: write-only custom role centered on `bigquery.tables.updateData` documented as the least-privilege path for pre-created tables; `roles/bigquery.dataEditor` documented as the convenient broader alternative; create permissions added only when the corresponding `auto_create` flag is on. |
| NFR5 | No plaintext credentials in config; secrets via env/ADC/workload identity; **Google credentials (ADC, service-account, impersonated) are never sent to custom/non-Google endpoints — prohibited outright, not merely off by default** (custom endpoints are emulator/dev paths using the explicit no-auth mode). |
| NFR6 | Contrib-equivalent CI (lint, unit ≥ 80% coverage, `metadata.yaml` + generated docs, lifecycle tests, golden-fixture schema tests, live-BigQuery integration tests) from the **first commit** — internally during M1 (no public artifacts), externally from the first external-repo commit / M2 publication. Live-BigQuery CI follows the §11.2 credential-isolation contract. |
| NFR7 | Cost transparency: published workload-based cost model covering stored bytes, Storage Write ingestion cost, dedup-query bytes scanned, and worked monthly examples for the reference workload. Cost comparisons vs alternatives are dated benchmarks, not categorical claims. |

### 5.4 Critical User Journeys (CUJs)

**CUJ-1 — Raw OTLP traces/logs in BigQuery for SQL analytics (P1, primary product contract)**
- *Precondition:* Any OTel pipeline; during external Alpha the user installs the custom collector distribution (documented one-liner OCB manifest / released binary or container image) — pure "add YAML and restart" onboarding is reserved for the milestone where the exporter ships in a published distribution; **the custom distribution remains the supported onboarding path through Beta unless the M5 contrib donation lands earlier**.
- *Steps:* (1) Deploy the distribution. (2) Configure `bigquery` exporter with project/dataset/tables; grant the SA the documented write-only role. (3) Run workload. (4) Query spans/logs with plain SQL; join to business tables; build dbt models / Looker dashboards using the published schema doc and dedup SQL.
- *Success:* Rows appear within seconds; every §7 schema field is populated per contract; the documented dedup query removes content-equivalent retry duplicates within the configured identity namespace (exact log identity only when **both** `source_record_namespace` and `source_record_id` are present; without a namespace, fingerprint fallback is table-wide — §7.4); no sampling imposed by the exporter.
- *Failure UX:* Missing write IAM → startup preflight warns best-effort; the **first real append fails authoritatively** with a message naming the expected permission and, when deterministically discoverable, the active principal (§9). When a generic ADC/WIF flow does not expose a reliable principal string, the message names the credential source class and destination instead of guessing. Confirmed-incompatible existing table schema → exporter refuses to start (health check red) rather than accepting traffic and dropping batches.

**CUJ-2 — Reliable delivery under outage (P2)**
- *Precondition:* CUJ-1 running with persistent `sending_queue` (`file_storage` on a restricted, size-bounded volume).
- *Steps:* (1) BigQuery unreachable / quota-exhausted for 10 min. (2) Collector queues and retries with backoff; self-telemetry shows queue depth, retry counts, and error classes (quota vs auth distinctly). (3) Service restores; queue drains.
- *Success:* **Zero** valid rows lost across the complete 600s reference outage under the published reference workload (batch size, queue bytes, volume headroom all stated — NFR3); duplicates on recovery bounded and removable via the published dedup SQL; queue-full behavior (backpressure/drop policy) matches documentation; clean shutdown drains in-flight appends within `write.shutdown_drain_timeout`, with durable restart replay promised only for a configured persistent queue.

**CUJ-3 — Redaction / governance pipeline (P3)**
- *Precondition:* Telemetry may contain sensitive data (log bodies, exception text, HTTP headers, DB statements, user identifiers).
- *Steps:* (1) Apply the shipped generic `transform`/`redaction` processor recipes covering bodies, resource/scope/record attributes, span events, links, exception data, headers, query text, and user identifiers. (2) Run the canary-secret test: inject a known marker secret separately as a value and as an attribute/`kvlist` key, then assert it reaches neither BigQuery nor exporter logs or diagnostic field paths.
- *Success:* Redaction is entirely in-pipeline with standard processors; exporter contributes FR13 (no raw rows in logs) and the canary test passes in CI.

**CUJ-4 — Multi-tenant routing (P2/P3)**
- *Precondition:* One collector serves many teams; tenant identity is deployment-controlled (e.g., per-receiver pipeline or authenticated headers), not a spoofable app attribute.
- *Steps:* `routingconnector` fans out to `bigquery/` instances, each targeting its own dataset with its own SA.
- *Success:* Dataset-level isolation with vanilla collector primitives; a tenant cannot write into another tenant's dataset by setting a resource attribute.

## 6. Ownership decision (Q1 — resolved): independent build in our own OSS repo

A substantial implementation exists as prior art: [contrib PR #46280](https://github.com/open-telemetry/opentelemetry-collector-contrib/pull/46280) (closed) and its external continuation by the same author — traces, logs, and metrics over Storage Write ingestion, used in production by the author's organization. Per §6.1, this document references only the PR description and publicly documented behavior, not that repository's source files.

**Decision: independent build, in a repository we own.** Rationale:

1. **Ownership requirement.** Release cadence, security response, support commitments, compatibility policy, and the eventual contrib-donation posture all require controlling the home repository rather than depending on a third-party maintainer's roadmap.
2. **Licensing.** OSPO/legal review determined that **reuse rights from the prior external implementation are unavailable**; the supporting evidence lives in the controlled inspection/provenance ledger (§6.1), not in this document. The implementation is therefore independent under the §6.1 protocol; the contrib PR description and OTLP specifications inform requirements as prior art only.
3. **Known contract gaps.** Its publicly documented representation (untagged-JSON attributes, stringified log bodies) is a legacy v0 schema incompatible with §7.3 regardless of ownership.

Hosting and approval status — stated precisely:

- **Intended** home: a versioned Go module in a repository under the GoogleCloudPlatform organization, Apache-2.0, repository name pending.
- **GoogleCloudPlatform placement is subject to Google's OSS release approval** ([release process](https://opensource.google/documentation/reference/releasing), [approval guidance](https://opensource.google/documentation/reference/releasing/approval)): GCP projects go through the GCP OSS process (Launcher2/OSPO) and must wait for approval before creating and publishing an external repository in a Google-managed org. Design completion does not imply that approval; the two are tracked separately.
- **Internal implementation preparation is unblocked once §6.1 provenance sign-off is recorded.** External repository creation and artifact publication wait for approval.
- **Courtesy outreach to @glacion is non-blocking**: inform them, credit PR #46280 as prior art in the eventual contrib donation ("similar components" section), and welcome their participation. A v0 compatibility view / migration guide is a candidate deliverable if users of the earlier exporter become a pilot cohort (non-blocking; the ownership decision is not reopened).
- The former M0 audit checklist (schema fidelity vs §7, managedwriter lifecycle, row-error recovery vs FR6, retry ownership vs FR5, request sizing, tests) is retained as a **requirements checklist for our implementation**, not as an adoption evaluation.

### 6.1 Independent implementation protocol (provenance)

"No code reuse" alone is not enforceable. The following protocol applies from before the first line of implementation code, handled conservatively with OSPO/legal guidance. (We use "independent implementation" rather than "clean-room" unless counsel confirms the latter characterization.)

1. **Source separation:** no direct links to, or reading of, the restricted prior external implementation's source files in design docs or implementation work. Permitted inputs: the contrib PR #46280 description, publicly documented behavior, official OTLP specifications, and independently written requirements (this document). The controlling non-reuse determination and its supporting repository metadata live only in the controlled ledger.
2. **Inspection ledger:** record who has already inspected the external implementation's source; where the approved protocol requires it, keep implementers isolated from that source.
3. **Provenance ledger:** maintain an origin record for important design choices (which decisions derive from OTLP specs, which from this doc's reviews, which from public prior-art behavior).
4. **Contributor hygiene:** provenance attestations for contributions, plus CLA/DCO as applicable.
5. **Pre-release checks:** license and code-similarity reviews before the first public release.
6. **Sign-off:** explicit OSPO/provenance approval recorded on this issue before internal implementation begins, and again before first publication.
7. **Pre-sign-off scrub:** before recording the first sign-off, audit this document and all linked material for direct references to the restricted prior external implementation (links, file names, code identifiers) and remove them; repository metadata needed for the record lives in the controlled inspection/provenance ledger, never in the implementation design. Neutral protocol terms such as “restricted prior external implementation” carry no public license-status assertion.

## 7. OTLP relational schema contract (versioned)

The schema is a **versioned, field-level contract** (`schema_version` published with the module; evolution rules: additive-only within a major, documented column modes, no type changes). Golden fixtures round-trip representative pdata values into expected BigQuery rows — including empty vs absent values, bytes bodies, nested attributes, span events, links, schema URLs, and logs with no event timestamp.

### 7.1 `otel_spans`

| Column | Type / mode | Source & rules |
|---|---|---|
| `start_timestamp` | TIMESTAMP REQUIRED (partition key, DAY) | span start (microsecond precision — BigQuery `TIMESTAMP` cannot hold nanos) |
| `start_time_unix_nano` | STRING REQUIRED | raw OTLP value, decimal string (covers the full unsigned `fixed64` domain) |
| `end_timestamp` | TIMESTAMP REQUIRED | span end (microsecond precision) |
| `end_time_unix_nano` | STRING REQUIRED | raw OTLP value, decimal string |
| `export_received_timestamp` | TIMESTAMP REQUIRED | exporter ingestion time; dedup tie-breaker |
| `duration_ns` | INT64 REQUIRED | end − start (nanoseconds; unit in the name) |
| `trace_id` | STRING(32) REQUIRED | lowercase hex |
| `span_id` | STRING(16) REQUIRED | lowercase hex |
| `parent_span_id` | STRING(16) | empty ≠ NULL: absent parent → NULL |
| `trace_state` | STRING | W3C tracestate |
| `flags` | INT64 | span flags |
| `name` | STRING REQUIRED | |
| `kind` | STRING | enum name (`SPAN_KIND_SERVER`, …) — enum-name encoding, not int |
| `status_code` | STRING | enum name (`STATUS_CODE_OK`, …) |
| `status_message` | STRING | |
| `service_name` | STRING | promoted from resource |
| `resource_attributes` | JSON | type-preserving encoding (§7.3) |
| `resource_schema_url` | STRING | |
| `resource_dropped_attributes_count` | INT64 | |
| `scope_name`, `scope_version` | STRING | |
| `scope_attributes` | JSON | |
| `scope_schema_url` | STRING | |
| `scope_dropped_attributes_count` | INT64 | `InstrumentationScope.dropped_attributes_count` |
| `schema_version` | STRING REQUIRED | this contract's version per row (§7.5); enables mixed-version tables and fail-fast checks |
| `span_attributes` | JSON | type-preserving encoding (§7.3) |
| `dropped_attributes_count`, `dropped_events_count`, `dropped_links_count` | INT64 | |
| `events` | REPEATED RECORD(name STRING, timestamp TIMESTAMP, time_unix_nano STRING, attributes JSON, dropped_attributes_count INT64) | raw nanos preserved per event |
| `links` | REPEATED RECORD(trace_id STRING, span_id STRING, trace_state STRING, flags INT64, attributes JSON, dropped_attributes_count INT64) | |

Default clustering: `service_name, name` (to be benchmarked in M3 against representative trace-analytics queries before being frozen).

### 7.2 `otel_logs`

| Column | Type / mode | Source & rules |
|---|---|---|
| `timestamp` | TIMESTAMP REQUIRED (partition key, DAY) | log `time_unix_nano`; when zero, falls back to `observed_time_unix_nano`; **when both are zero (OTLP permits this) falls back to `export_received_timestamp`**, so the partition key is always populated. Fallback recorded in `timestamp_source`; original zero raw values preserved in the `*_unix_nano` columns |
| `timestamp_source` | STRING | `event` \| `observed` \| `exporter` |
| `time_unix_nano` | STRING REQUIRED | raw OTLP value, decimal string (may be `"0"`) |
| `observed_timestamp` | TIMESTAMP | |
| `observed_time_unix_nano` | STRING REQUIRED | raw OTLP value, decimal string (may be `"0"`) |
| `export_received_timestamp` | TIMESTAMP REQUIRED | exporter ingestion time |
| `trace_id` | STRING(32) | NULL when unset |
| `span_id` | STRING(16) | NULL when unset |
| `flags` | INT64 | log record flags |
| `event_name` | STRING | distinguishes OTel Events represented as LogRecords |
| `severity_number` | INT64 | |
| `severity_text` | STRING | |
| `body` | JSON | type-preserving `AnyValue` encoding (§7.3) — plain JSON would lose type information |
| `service_name` | STRING | promoted from resource |
| `resource_attributes`, `scope_attributes`, `log_attributes` | JSON | §7.3 |
| `resource_schema_url`, `scope_schema_url` | STRING | |
| `scope_name`, `scope_version` | STRING | |
| `resource_dropped_attributes_count`, `scope_dropped_attributes_count`, `dropped_attributes_count` | INT64 | |
| `schema_version` | STRING REQUIRED | this contract's version per row (§7.5) |
| `source_record_namespace` | STRING NULLABLE | trusted namespace for `source_record_id`, from deployment-controlled config (§7.4); NULL when no namespace configured |
| `source_record_id` | STRING NULLABLE | stable source record identity, populated from a configurable record attribute when the instrumentation supplies one; NULL otherwise |
| `record_fingerprint` | STRING REQUIRED | versioned content hash computed by the exporter (§7.4) |

Logs get **no default clustering** in v1 — this is **final**, not pending (log query patterns vary too widely for a useful default). Only the span clustering default remains an M3 benchmark candidate.

### 7.3 Type-preserving `AnyValue` encoding — **`otlp-protojson-v1`** (Q2: resolved)

OTLP attribute values and log bodies are `AnyValue` (string, bool, int64, double, bytes, array, kvlist, empty). Plain JSON cannot distinguish int64 vs double at scale boundaries, loses bytes, and conflates empty/null. The contract is **`otlp-protojson-v1`**, following the canonical [OTLP `AnyValue` definition](https://github.com/open-telemetry/opentelemetry-proto/blob/main/opentelemetry/proto/common/v1/common.proto) and [ProtoJSON encoding rules](https://protobuf.dev/programming-guides/json/):

- The OTLP `AnyValue` oneof field name is the type tag, in original snake-case proto names: `string_value`, `bool_value`, `int_value`, `double_value`, `bytes_value`, `array_value`, `kvlist_value`.
- ProtoJSON scalar rules: int64 as decimal **strings** (survives JSON number limits), bytes as padded base64, special doubles as `"NaN"` / `"Infinity"` / `"-Infinity"`.
- SQL `NULL` is reserved for an absent column; an explicitly empty `AnyValue` encodes as `{}`.
- Attribute bags encode as an object mapping attribute keys to encoded `AnyValue`s (OTLP requires unique keys).
- The encoding version is published with the table schema; extraction UDFs ship for common lookups.

Examples:

```json
{"string_value": "hello"}
{"int_value": "9223372036854775807"}
{"bytes_value": "AQIDBA=="}
{"array_value": {"values": [{"string_value": "one"}, {"int_value": "2"}]}}
{"kvlist_value": {"values": [{"key": "enabled", "value": {"bool_value": true}}]}}
```

Attribute bag:

```json
{"http.status_code": {"int_value": "200"}, "cache.hit": {"bool_value": true}}
```

**Relation to prior art:** the earlier exporter's publicly documented representation (untagged-JSON attributes, stringified log bodies — as described in contrib PR #46280 and its public documentation; per the §6.1 protocol we do not link to or read its source files) loses the fidelity this contract promises. It is treated as a **legacy v0 schema**; the resolved decision is an independent build (§6), and a v0 compatibility view/migration guide is an optional pilot enabler, not a contract requirement.

**Unknown `AnyValue` variants:** the exporter operates on pdata, and [`pcommon.Value`](https://pkg.go.dev/go.opentelemetry.io/collector/pdata/pcommon) exposes only the compiled value kinds (empty, string, int, double, bool, map, slice, bytes) — a future unknown oneof variant never reaches the exporter with its proto field name, so no "unsupported variant" encoding is promised at this boundary. Contract: the supported Collector/pdata/OTLP baseline is pinned (§7.5); a newly introduced `AnyValue` variant requires a Collector/protobuf upgrade, and the OTLP receiver may reject or ignore it before pdata exists. Behavior for such payloads is tested at the **receiver boundary**, not the exporter.

### 7.4 Identity & deduplication (Q4: resolved)

**Spans:** identity = (`trace_id`, `span_id`) — dedup SQL orders on the raw nanos with an ingestion-time tie-breaker (microsecond `end_timestamp` alone can tie across retries):

```sql
SELECT * EXCEPT (rn)
FROM (
SELECT
*,
ROW_NUMBER() OVER (
PARTITION BY trace_id, span_id
ORDER BY
CAST(end_time_unix_nano AS BIGNUMERIC) DESC,
export_received_timestamp DESC
) AS rn
FROM `PROJECT.DATASET.otel_spans`
)
WHERE rn = 1
```

**Logs:** two-tier identity.

- Exact identity is the **typed tuple (`source_record_namespace`, `source_record_id`)**. `source_record_id` is populated from a configurable record attribute when instrumentation provides one. `source_record_namespace` comes from **authenticated or deployment-controlled identity** (static exporter config, per-tenant pipeline/receiver identity) — never from an arbitrary telemetry attribute, because a bare locally-unique ID can collapse unrelated records across services, and an untrusted producer could deliberately collide with another record's ID. Deployment control establishes *trust*; uniqueness requires more: the namespace is defined as the **globally stable identifier-authority scope across every writer targeting the table** — two deployments whose IDs come from different authorities must not share a namespace value. A `source_record_id` arriving with no configured namespace is stored but **not used for exact identity**. When exact identity is unavailable, a configured namespace still scopes fingerprint deduplication; this deliberately prevents identical content from unrelated tenants/authorities from collapsing. With no namespace, fingerprint fallback is table-wide. **Configuration grammar:** the empty string means unset and is written as SQL `NULL`; a nonempty namespace is case-sensitive, receives no Unicode/case normalization, is 1–128 ASCII bytes, and matches `^[A-Za-z0-9][A-Za-z0-9._:/-]{0,127}$`; invalid configuration fails startup. A namespace change is an **identity migration** and must roll out fleet-atomically for all writers targeting a table (drain old writers before enabling the new value); mixed old/new namespaces create a documented duplicate window and are not a supported rolling change. **Attribute conversion:** only a non-empty *string* attribute value is accepted as `source_record_id`; absent, empty, bytes, integer, boolean, or oversized values yield `source_record_id = NULL`, fingerprint identity applies, and `otelcol_bigquery_source_record_id_fallbacks` increments for an invalid present value or a valid ID with no namespace (`namespace_unset`) — the otherwise-valid log row is never rejected for this. Absence of this optional attribute is normal and does not increment the counter.
- `record_fingerprint` (required): computed by the exporter at serialization time:

```text
record_fingerprint = "fp1:" + lowercase_hex(SHA256(canonical_log_record))
```

The `fp1:` prefix is an independent fingerprint-algorithm namespace (deliberately not `v1`, which would be mistaken for schema v1 while the first schema is `v0alpha1`), and it is **outside the digest** — prepended to the hex output, never part of the hash input. Both facts are frozen in the golden vectors.

`canonical_log_record` includes all non-derived OTLP source fields: original event and observed timestamps; trace ID, span ID, and flags; event name; severity number and text; body; log attributes and dropped count; **resource attributes**, resource schema URL, and dropped count; scope name, version, attributes, schema URL, and scope dropped-attributes count. (Resource attributes are included deliberately — scope-plus-record-only hashing would collapse identical messages from different services, pods, or tenants.) Derived projections (`service_name`, partition `timestamp`, `timestamp_source`, `source_record_namespace`, `source_record_id`, `record_fingerprint`, `schema_version`, and `export_received_timestamp`) are excluded; `source_record_id` remains represented inside `attributes` when it originated there.

The pre-JCS root value is frozen — JCS canonicalizes this exact value rather than choosing its structure:

```json
{
"time_unix_nano": "",
"observed_time_unix_nano": "",
"trace_id": null,
"span_id": null,
"flags": 0,
"event_name": "",
"severity_number": 0,
"severity_text": "",
"body": {},
"attributes": {},
"dropped_attributes_count": 0,
"resource": {
"attributes": {},
"schema_url": "",
"dropped_attributes_count": 0
},
"scope": {
"name": "",
"version": "",
"attributes": {},
"schema_url": "",
"dropped_attributes_count": 0
}
}
```

Every key above is always present. Empty IDs are JSON `null`; nonempty IDs are fixed-length lowercase-hex strings. For a log with one-sided trace context, the nonempty raw ID remains in this fingerprint root while the other ID is `null`, even though both relational ID projections are `NULL`; this prevents that distinct source content from collapsing silently. Empty strings and zero numeric source fields stay explicit; timestamps are unsigned decimal strings; an empty body is the §7.3 empty `AnyValue` `{}`; empty attribute maps are `{}`. Attribute values/body use §7.3 recursively, and array order is preserved. This presence/default grammar is part of `fp1`, not an implementation detail, and has a golden vector.

Canonicalization is a **byte-level contract**, not a semantic one: the exporter constructs the exact root value above with tagged values per §7.3, then canonicalizes it with **[RFC 8785 JCS](https://www.rfc-editor.org/rfc/rfc8785.html)** — JCS's defined property ordering, string escaping, and number rendering govern (superseding any other key-ordering claim); NaN and infinities remain tagged *strings* before canonicalization, so JCS never sees a non-finite number. The implementation must **never hash a protobuf library's marshaled ProtoJSON output** — protojson output is explicitly unstable across builds. IDs and timestamps normalized (raw `*_unix_nano` values, including zeros). Published **golden vectors** (input → exact pre-JCS value → canonical bytes → SHA-256) ship with the schema and run in CI against **at least two protobuf/Go dependency versions** — a canonicalization-stability compatibility test, not a widened support matrix; M1's dependency locks remain the single officially supported set.

- Canonical dedup SQL partitions by **separate expressions** — identity kind, namespace, and ID-or-fingerprint — never by string concatenation (a `CONCAT(namespace, '/', id)` flattening would collide `('a/b','c')` with `('a','b/c')` and mix source IDs with fingerprints in one string domain):

The canonical query is **complete and executable** (not a `PARTITION BY` fragment — pilots must all get the same result), retaining the latest-ingested copy:

```sql
SELECT * EXCEPT (rn)
FROM (
SELECT
*,
ROW_NUMBER() OVER (
PARTITION BY
(source_record_namespace IS NOT NULL AND source_record_id IS NOT NULL), -- identity kind
IFNULL(source_record_namespace, ''),
IF(source_record_namespace IS NOT NULL AND source_record_id IS NOT NULL,
source_record_id, record_fingerprint)
ORDER BY export_received_timestamp DESC
) AS rn
FROM `PROJECT.DATASET.otel_logs`
)
WHERE rn = 1
```

**Contract wording (deliberately scoped):** canonical SQL removes **content-equivalent retry duplicates within the configured identity namespace**. When no namespace is configured, fingerprint fallback is table-wide. A content fingerprint cannot distinguish two legitimately emitted logs whose complete contents are identical inside that scope — exact record identity is available only via the trusted (`source_record_namespace`, `source_record_id`) tuple. Without it, the namespace-plus-fingerprint key is explicitly best-effort content deduplication, not a globally unique log identity.

### 7.5 Schema baseline & evolution policy

- **Pinned baselines:** the initial release family pins an exact Collector release and OTLP proto version; both are recorded in the schema doc and the release notes.
- **Stable fields only:** the initial release family maps fields that are Stable in the pinned OTLP protos. Development/Alpha fields (e.g., `Resource.entity_refs`) are **not mapped**; when they stabilize they enter through the additive process below. Profiles-only Alpha `AnyValue` variants in trace/log input follow the unsupported-variant rule in §7.3.
- **Additive evolution:** new OTLP fields enter the BigQuery schema as NULLABLE columns in a minor `schema_version` bump; no type changes or column removals within a major.
- **`schema_version` exposure:** required per-row STRING column on both tables (the writer-version record; supports mixed-version tables during rollout). The table **label** is a **control-plane field** maintained by DDL/Terraform or `auto_create` — the data-plane exporter never patches it (label updates require `bigquery.tables.update`, which the write-only role intentionally lacks, and a "newest writer" label would flap during mixed-version rollout/rollback).
- **Alpha compatibility epochs** (a bare `v0` major would let an incompatible `v0alpha2` pass a major-only startup check and write a different row shape into a `v0alpha1` table): during Alpha, the table label carries the **exact compatibility epoch** (`v0alpha1`, `v0alpha2`, …). Every incompatible Alpha correction creates a **new epoch**; writing a new epoch requires a **new table or an explicit recorded migration**; mixed writer versions are permitted only for changes declared compatible/additive within an epoch. Rollout has **three distinct procedures — a blanket "DDL first, then exporters" rule is unsafe for incompatible changes** (it would expose old writers to incompatible in-place DDL, violating the mixed-writer prohibition above):
1. *Additive, same-table:* apply additive DDL first (NULLABLE columns), then roll exporters; old and new writers coexist; rollback = roll exporters back (added columns stay NULL).
2. *Incompatible, new-table cutover (the default for epoch bumps):* create the new table with the new epoch label, cut exporters over, optionally expose a union/compat view and backfill the old data; rollback = point exporters back at the old table, which was never modified.
3. *In-place migration (only if a change explicitly supports it):* quiesce old writers, drain in-flight appends, apply DDL, then start new writers — never while old writers run.

After the v1 freeze, labels switch to major-only. This policy binds **M1 startup validation and golden schemas** — it is not deferred to the M2 support package.
- **Label grammar (M2 users pre-create tables by hand, so "missing label" is a real state):** the exact label key is `otel_bq_schema_epoch`; values match `v0alpha[0-9]+` during Alpha and `v[0-9]+` post-freeze. A **missing or unparseable label is a startup rejection** (fail-fast, named remediation), exactly like a declared-incompatible one. The released DDL/Terraform sets the label; it is not optional documentation.
- **Fail-fast:** at startup, the exporter refuses to write to a table whose declared compatibility epoch (Alpha) or major (post-freeze) is incompatible with its own; incompatibility is a startup error, not a silent write.
- **Stability lifecycle:** M2 publishes the schema as **`v0alpha1`** — documented breaking corrections are allowed during Alpha (each incompatible correction creates a new compatibility epoch, above). **v1 freezes only after at least two independent design partners complete representative workflows** (SQL analytics, deduplication, extraction UDFs, dbt/Looker models, and a version upgrade) against the released artifact. The additive-only evolution rules above bind within a frozen major; freezing v1 before that validation would be an accepted compatibility-cost tradeoff requiring an explicit recorded decision, and is not the default plan.

### 7.6 Record-validation matrix (invalid identifiers & timestamps)

At the exporter boundary, [`pcommon.Timestamp`](https://pkg.go.dev/go.opentelemetry.io/collector/pdata/pcommon#Timestamp) is unsigned nanoseconds since the Unix epoch, while `TraceID` and `SpanID` are already fixed `[16]byte` and `[8]byte` pdata values. The complete timestamp domain runs from 1970 through approximately `2554-07-21T23:34:33.709551615Z`, inside BigQuery `TIMESTAMP`'s year-9999 limit. The exporter **must not** call pdata `Timestamp.AsTime()` for conversion because its current implementation casts the nanoseconds to signed `int64` and wraps values above `math.MaxInt64`. It instead computes `seconds = uint64_nanos / 1e9` and `nanos = uint64_nanos % 1e9`, then writes the BigQuery value truncated (never rounded) to microseconds; the decimal raw column preserves all nanoseconds. The initial-release-family policy:

| Condition | Policy |
|---|---|
| Span with zero start or end time | **Permanently reject** (malformed span), reason `invalid_timestamp` |
| Any nonzero span/log timestamp, including maximum `uint64` | Convert with the unsigned quotient/remainder algorithm above; never via `AsTime()`; preserve the raw decimal string |
| Log with zero `time_unix_nano` but nonzero `observed_time_unix_nano` | Partition timestamp from **observed** time (`timestamp_source: observed`) — per the OTel Logs data model, not an exporter fallback |
| Log with **both** timestamps zero | Exporter fallback per §7.2 (`timestamp_source: exporter`) — the only exporter-time fallback |
| Span event with zero timestamp | **Drop the event element only** (span kept), reason `invalid_event_timestamp`; every nonzero `uint64` event timestamp is convertible |
| Link with zero trace or span ID | **Drop the link element only** (span kept), reason `invalid_link_id` |
| Zero/absent parent span ID | Valid root span — `parent_span_id` NULL, no rejection |
| Log with partial trace context (one of trace/span ID nonzero, the other zero) | Keep the row; **NULL both** `trace_id` and `span_id` projections (raw context is not reconstructible), reason `partial_trace_context` |
| `end < start` | **Permanently reject**, reason `negative_duration` |
| `duration_ns` overflow of signed INT64 | **Permanently reject**, reason `duration_overflow` |
| Missing / zero trace ID or span ID on a span | **Permanently reject**, reason `invalid_id` |
| Zero/absent trace/span IDs on a log | Allowed — columns NULL (logs are valid without trace context) |
| Invalid UTF-8 in any string or map key | **Permanently reject the enclosing record before fingerprinting or serialization**, reason `invalid_utf8`; never replace with U+FFFD |

Every permanently rejected row increments bounded `otelcol_bigquery_rejected_rows{signal,reason}`; each dropped event/link increments FR12a `otelcol_bigquery_dropped_span_elements{element_kind,reason}`; each kept log with one-sided context increments `otelcol_bigquery_partial_trace_contexts`. The source OTLP `dropped_events_count` and `dropped_links_count` columns remain unchanged — exporter-side drops never rewrite source counters. Golden exporter fixtures are required for **every row above**, including invalid UTF-8 in a string and map key, one-sided trace context (raw nonzero ID included in `fp1`, relational projections both `NULL`), zero/reversed times, duration overflow, and maximum `uint64`; the maximum value must succeed and produce `2554-07-21 23:34:33.709551 UTC` in the microsecond column while preserving `"18446744073709551615"` raw. Malformed wire-length IDs are tested at the **OTLP receiver boundary**, because wrong lengths cannot exist in pdata; exporter fixtures test all-zero IDs only.

## 8. Storage Write API design

### 8.1 Append pipeline

- One `ManagedStream` (default stream) per destination table, shared across consumers; recreated on unrecoverable stream errors with backoff.
- `AppendRows` is asynchronous; waiting on each result inline would serialize the path. Design: stream flow control uses **managedwriter's native in-flight limits**, with the exporter **explicitly setting both** `WithMaxInflightRequests` (default 8) and `WithMaxInflightBytes` (default 64 MiB) — these are *exporter* defaults, not library defaults: managedwriter's own defaults are 1,000 requests and **unlimited** bytes, so relying on them would void the bounded-memory contract. **These limits are per writer (per `ManagedStream`), not per exporter instance** — the config names them `per_stream_*`, and the documented worst-case exporter budget is `per_stream limit × active destination tables` (traces + logs = 2× by default: up to 16 in-flight requests / 128 MiB, before retained pdata, encoded buffers, and queue memory). **Multiplexing is disabled for the initial release family** (per-writer limits are ignored on multiplexed streams). A **saturation test exercises both signals simultaneously** and proves producer backpressure engages before the aggregate bound is exceeded. The exporter keeps a reaper solely to **map append acknowledgements back to Collector batches**, acknowledging a batch only when **all** its appends are acknowledged.
- The exporterhelper per-attempt `timeout` covers the complete backend attempt, including waiting for a managedwriter flow-control slot and the append result. The reference timeout must exceed measured p99 slot-wait-plus-append latency with headroom. The dual-signal saturation test therefore also asserts that local flow control at the reference load creates **no timeout-induced retryable errors or replay amplification**; a failure means the timeout/in-flight defaults or throughput claim must be changed, not waived.
- **Shutdown has its own budget:** exporterhelper `timeout` remains the per-attempt backend timeout; it is not reused as a drain deadline. On shutdown the exporter waits up to `write.shutdown_drain_timeout` for in-flight acknowledgements. If the deadline expires, active requests terminate retryably. With a configured persistent queue, the original unacknowledged queue item remains durable for restart replay; an in-memory queue or disabled queue cannot survive process exit and therefore offers best-effort shutdown only. A SIGTERM/crash-and-restart integration test (M2 exit criterion) validates the persistent case; a separate memory/no-queue test proves the documented non-durable outcome. If the persistent test cannot be made to pass for the pinned Collector version, the doc and README must downgrade the claim to best-effort — silently keeping the stronger wording is not an option.
- Two-stage size enforcement: FR8 structural limits (depth, counts, value sizes) run pre-serialization; row-byte checks are exact post-protobuf-serialization; the request bound is a **payload-sum bound** per FR7 (the final request including envelope/schema overhead is constructed inside [managedwriter](https://pkg.go.dev/cloud.google.com/go/bigquery/storage/managedwriter) and is not observable at the public API), with the default 8 MiB headroom below the <10 MB API limit and split-and-retry on unexpected server oversize rejections. Oversized individual rows are FR6-style permanent rejections.

### 8.2 Retry ownership

Exactly one owner per class, configured explicitly: the exporter sets **`managedwriter.EnableWriteRetries(false)`** (explicit even though write retries are already the default-off behavior — the contract must not depend on a library default) — managedwriter owns **gRPC connection recovery** (the Go client automatically reconnects closed connections; the default stream always exists and needs no creation) and nothing else; exporterhelper owns **append replay** (backoff + jitter, honoring `retry_on_failure`). This split prevents multiplied attempts and keeps duplicate amplification bounded. There is no generic "invalid stream" class — stream/connection failures follow the exact taxonomy in §8.3, each branch with its own fault test.

### 8.3 Error matrix (initial-release-family contract, abridged)

**Ownership is defined by stage, not by error name:** managedwriter owns **connection reopening only** (the default stream needs no creation; the Go client reconnects closed connections — but reconnecting transport is *not* replaying an unresolved append); exporterhelper is the **sole append-replay owner**. The standard helper path has no fourth “block health and retain forever” disposition: a request is either successful, permanently rejected/dropped and counted, or returned retryable. Retryable requests make the component unhealthy while they fail, but remain durable only until the configured queue/retry boundary; at `max_elapsed_time` expiry exporterhelper terminates the request and the persistent queue deletes it, as NFR3 already states. **Phase rule:** a confirmed schema/location/label mismatch discovered by startup validation fails `Start` before the exporter accepts telemetry. A destination-state failure first observable on an append (including permission loss) never makes otherwise-valid telemetry permanently malformed: it is returned retryable for the configured repair window, whether it occurs before or after the first acknowledged append. Every matrix row below is scoped to that contract.

| Failure | Class | Owner | Outcome |
|---|---|---|---|
| Row fails serialization / FR8 structural validation / post-serialization row-byte limit | Permanent (row) | exporter | permanently reject/drop row, metadata-only diagnostic, count `otelcol_bigquery_rejected_rows{signal,reason}` |
| `AppendRows` row errors | Permanent (rows) — **whole request not appended** | exporter | drop+count invalid rows internally, append valid subset, return `nil` on success; only retryable un-acked telemetry ever goes back via `consumererror` (FR6 contract test) |
| gRPC connection closure/reset | Retryable | managedwriter reconnects; unresolved appends → exporterhelper | append is never classified permanent merely because of a transport event; it remains retryable or uncertain-ack, but can still be lost if the overall retry horizon expires |
| Terminal `ManagedStream` wrapper failure | Retryable (after rebuild) | exporter rebuilds wrapper; replay via exporterhelper | still-unacknowledged telemetry returned for replay |
| Locally generated malformed descriptor / impossible schema state | Permanent invariant/configuration failure | exporter | reject before append where possible; otherwise fail the batch and health; this class means the exporter build/config is wrong, not that an operator changed the destination |
| Destination schema/partition/epoch drift discovered after startup | Retryable destination state | exporterhelper | component unhealthy; return unacknowledged telemetry through the 900s repair window; revalidate before replay; horizon expiry drops/counts the queue item |
| `TABLE_NOT_FOUND` after startup | Retryable destination state | exporterhelper | component unhealthy; authorized control-plane repair may recreate + revalidate; replay through the repair window; horizon expiry drops/counts the queue item |
| Default-stream `STREAM_NOT_FOUND` / invalid-state responses | Invariant violation — **Retryable after one bounded wrapper rebuild** | exporter rebuilds once; recurrence → exporterhelper | one rebuild+retry, then recurrence is returned retryable and the component remains unhealthy; exact-`StorageError`-code diagnostics (`STREAM_NOT_FOUND`, `INVALID_STREAM_TYPE`, `INVALID_STREAM_STATE`); retry-horizon expiry drops/counts the queue item per NFR3 rather than claiming indefinite retention |
| `RESOURCE_EXHAUSTED` (quota) | Retryable | exporterhelper | backoff; distinct metric label |
| `UNAVAILABLE` / `INTERNAL` append-RPC results | Retryable | exporterhelper | telemetry returned; backoff (transport reconnection itself belongs to the gRPC row above) |
| `PERMISSION_DENIED` on an append | Retryable destination state | exporterhelper | component unhealthy; IAM may be repaired in place; name expected permission and configured principal when deterministically known, otherwise credential source class + destination (§9); horizon expiry drops/counts the queue item |
| Confirmed non-refreshable credential/configuration error before append | Permanent startup/configuration failure | exporter | fail startup; never accept and then row-drop telemetry for it |
| Credential refresh failure | Retryable (bounded by `max_elapsed_time`) | auth layer refreshes; **the affected append surfaces to exporterhelper as a terminal retryable result** | telemetry returned for replay and component unhealthy while failing; horizon expiry drops/counts the queue item per NFR3 |
| Uncertain ack (deadline mid-append) | Retryable | exporterhelper | retry ⇒ possible duplicates — covered by the at-least-once contract + §7.4 dedup |
| `write.shutdown_drain_timeout` with in-flight appends | Retryable shutdown outcome | exporter | with persistent storage, the original unacknowledged queue item remains for restart replay; memory/no queue is non-durable and explicitly best effort |

### 8.4 Delivery contract

**Default-stream delivery is at least once and may duplicate data.** This sentence appears verbatim in the README, config comments, and schema doc, alongside §7.4 dedup guidance.

"Bounded duplicates" is a falsifiable invariant, not a vibe:

> For **one accepted exporterhelper queue item, processed by one exporter instance, with `N` uncertain-acknowledgement retries within one uninterrupted in-memory retry loop**, a logical record has **at most `N+1` physical copies**, with no hidden retry multiplier from managedwriter (§8.2 disables its write retries).

Explicitly **outside** the invariant: upstream SDK retries, pipeline fanout, multiple Collectors exporting the same telemetry, and persistent-queue replay of an original item after a crash (FR6 durability scope). Crash replays are counted and tested **separately** (the M2 crash-replay case); `otelcol_bigquery_uncertain_ack_replays` does not observe them.

Enforcement:

- `retry_on_failure.max_elapsed_time` is finite (reference config: 900s — deliberately **above** the CUJ-2 ten-minute outage, since horizon expiry is a terminal error after which the persistent queue deletes the item; an unbounded horizon would make the duplicate bound meaningless, a sub-outage horizon would break the no-loss promise). The docs publish the **operational worst-case `N`** implied by the reference policy (900s horizon, 1s initial / 30s max backoff), so operators can translate the invariant into a concrete physical-copy ceiling.
- Metric `otelcol_bigquery_uncertain_ack_replays` counts in-loop replays triggered by uncertain acknowledgements.
- The M2 fault-injection test injects `N` uncertain-ack replays and **asserts the physical-row count** per logical record is ≤ `N+1`.
- Documentation states the bound assumes exactly one replay owner (exporterhelper, per §8.2).

### 8.5 No DLQ in the initial release family — "reject" means drop

The initial release family (`v0alpha1` through the v1 freeze) defines no dead-letter destination. Rows that fail permanently (serialization, structural limits, row-byte limit, `AppendRows` row errors) are **permanently rejected and dropped**: counted in `otelcol_bigquery_rejected_rows{signal,reason}` with metadata-only diagnostics, then discarded. A real quarantine/DLQ table or file — with its own destination, retention, security, and retry contract plus IAM surface — is a separate future feature, not an implied behavior of this family.

### 8.6 Self-telemetry ownership and exact metric names

The pinned Collector's exporterhelper owns the standard `otelcol_exporter_*` telemetry. The component must not redeclare it:

| Owner | Exact emitted metric | Contract |
|---|---|---|
| exporterhelper | `otelcol_exporter_queue_size`, `otelcol_exporter_queue_capacity`, `otelcol_exporter_in_flight_requests` | queue/in-flight gauges |
| exporterhelper | `otelcol_exporter_enqueue_failed_spans`, `otelcol_exporter_enqueue_failed_log_records` | queue enqueue failures/drops by signal |
| exporterhelper | `otelcol_exporter_send_failed_spans`, `otelcol_exporter_send_failed_log_records` | standard failed-send item counters |
| exporterhelper | `otelcol_exporter_sent_spans`, `otelcol_exporter_sent_log_records` | standard successful-consume counters; **not** the authoritative BigQuery row-ack counter because the component can return success after internally filtering permanent rows |

The bigquery exporter owns these bounded-cardinality `metadata.yaml` metrics (current mdatagen convention is `otelcol__`, so component type `bigquery` correctly emits the `otelcol_bigquery_*` names below; no extra `exporter` segment is inserted):

| Metadata key → exact emitted metric | Type / unit | Bounded attributes and increment point |
|---|---|---|
| `acknowledged_rows` → `otelcol_bigquery_acknowledged_rows` | monotonic sum / `{rows}` | `signal={traces,logs}`; increment only after BigQuery acknowledgement |
| `rejected_rows` → `otelcol_bigquery_rejected_rows` | monotonic sum / `{rows}` | `signal`; `reason={serialization_error,structural_limit,row_too_large,append_row_error,invalid_timestamp,negative_duration,duration_overflow,invalid_id,invalid_utf8}` |
| `dropped_span_elements` → `otelcol_bigquery_dropped_span_elements` | monotonic sum / `{elements}` | `element_kind={event,link}`; `reason={invalid_event_timestamp,invalid_link_id}` |
| `source_record_id_fallbacks` → `otelcol_bigquery_source_record_id_fallbacks` | monotonic sum / `{records}` | `reason={empty,non_string,oversized,namespace_unset}`; absent optional IDs do not increment |
| `partial_trace_contexts` → `otelcol_bigquery_partial_trace_contexts` | monotonic sum / `{records}` | no telemetry-derived attributes; increment when one-sided context is kept with both projections `NULL` |
| `uncertain_ack_replays` → `otelcol_bigquery_uncertain_ack_replays` | monotonic sum / `{replays}` | `signal`; in-loop uncertain-ack replay only, not crash replay |
| `retryable_append_errors` → `otelcol_bigquery_retryable_append_errors` | monotonic sum / `{attempts}` | `signal`; `class={quota,unavailable,internal,transport,credential_refresh,destination_state,invariant,uncertain_ack}` when a retryable result is returned to exporterhelper |

An M1 generated-metadata test asserts the exact names, units, types, and complete bounded attribute sets. No metric label contains a destination name, error message, telemetry value, record ID, or fingerprint.

## 9. Table management & IAM (control plane vs data plane)

- **Data plane (default):** pre-created tables; documented write-only custom role centered on `bigquery.tables.updateData` (+ `bigquery.tables.get` for validation) — the least-privilege path. `roles/bigquery.dataEditor` documented as the convenient broader role, not as "least privilege."
- **Control plane (opt-in):** `auto_create.tables` adds table create/metadata permissions; `auto_create.dataset` adds project-level dataset creation. Each flag's incremental permissions documented separately.
- **Startup validation:** exporter validates destination schema compatibility, dataset location, and partition/clustering config at startup; on **confirmed** incompatibility it fails startup / reports unhealthy rather than accepting traffic and permanently dropping batches. Concurrent-create races resolved by treating `ALREADY_EXISTS` as success-then-validate. Note: schema validation requires metadata access (`bigquery.tables.get`) beyond a strictly write-only role — documented as part of the recommended custom role.
- **Permission preflight is best-effort, not authoritative:** BigQuery documents that `tables.testIamPermissions` is not intended for authorization checking and may fail open, and `tables.get` proves metadata access, not `bigquery.tables.updateData`. Contract: startup preflight warns on likely problems; startup **fails only on confirmed missing permission**; the **first real append is the authoritative write check**, and its failure message names the expected permission. It names the active principal only when deterministically available from the configured impersonation/service-account identity; generic ADC/WIF flows may not expose a reliable principal string, so those diagnostics name the credential source class and destination rather than guessing or making a network identity-discovery call.
- **Credentials:** workload identity / ADC is the production default. Service-account key files are discouraged compatibility support: accepted **by path only** (never inline JSON or environment-carried credential bodies), with documented restrictive mount permissions, credential precedence order, reload/rotation behavior, and the IAM permissions required for impersonation.
- **Destination governance — M2 minimum profile** (the destination must not be governed more weakly than the queue/runtime at first public Alpha): documented retention/partition-expiration guidance, reader-IAM patterns, dataset residency/location statement, and CMEK compatibility notes ship **with the M2 release**. The full governance guide (deletion workflows, audit logging depth) completes in M3. For M3 `auto_create` (FR11b): CMEK and partition expiration are **explicit configuration with mismatch validation** — never silently inherited project defaults.
- **Persistent queue security:** restricted directory permissions, encrypted volume recommendation, bounded size, cleanup/retention on decommission, and separation by trust domain when one host serves multiple tenants.

## 10. Example configuration

```yaml
extensions:
file_storage:
directory: /var/lib/otelcol/bigquery-queue # restricted directory on an encrypted, size-bounded volume (§11.3)

receivers:
otlp:
protocols:
grpc:
endpoint: 127.0.0.1:4317 # loopback default; document TLS/auth before network exposure

processors:
batch:
timeout: 1s # explicit: queue-capacity arithmetic depends on this flush cadence (NFR3)
send_batch_size: 8192 # trigger size; scoped independently to each signal pipeline
send_batch_max_size: 8192 # hard maximum Collector batch size

exporters:
bigquery:
project: my-gcp-project
dataset: telemetry
location: US
traces:
table: otel_spans
logs:
table: otel_logs
source_record_id_attribute: "" # optional record attribute carrying a stable source record ID (§7.4)
source_record_namespace: "" # empty = SQL NULL; otherwise 1–128 ASCII bytes matching §7.4 grammar;
# an identity migration requires a fleet-atomic rollout
limits: # FR8 structural limits (pre-serialization)
max_nesting_depth: 16
max_attribute_count: 512
max_events_per_span: 256
write: # FR7 encoded-size limits (post-serialization, pre-AppendRows)
max_row_bytes: 1048576
max_request_bytes: 8388608 # headroom bound on summed row payloads, below the <10 MB API limit (§8.1)
per_stream_max_inflight_requests: 8 # PER destination table/stream (§8.1); explicitly overrides
# managedwriter's own default (1000)
per_stream_max_inflight_bytes: 67108864 # PER destination table/stream; managedwriter default is
# unlimited (0). Worst-case exporter total = value × tables
shutdown_drain_timeout: 30s # exporter shutdown budget; distinct from exporterhelper per-attempt timeout
auto_create:
dataset: false
tables: false # default: pre-created tables + write-only role
auth:
credentials_file: "" # empty → ADC / workload identity (production default); key files are
# discouraged compatibility support, accepted by path only (§9)
impersonate_service_account: ""
sending_queue:
enabled: true
storage: file_storage
sizer: requests # explicit: queue_size is a Collector request/batch count, not rows or bytes
queue_size: 5000 # batches, not rows/bytes
retry_on_failure:
enabled: true
initial_interval: 1s
max_interval: 30s
max_elapsed_time: 900s # finite by contract (§8.4); must exceed the outage you intend to survive —
# horizon expiry is terminal and the queued item is then deleted (NFR3)
timeout: 15s # exporterhelper timeout per individual backend attempt

service:
extensions: [file_storage]
pipelines:
traces: { receivers: [otlp], processors: [batch], exporters: [bigquery] }
logs: { receivers: [otlp], processors: [batch], exporters: [bigquery] }
```

## 11. Delivery & upstreaming plan (external-first)

Contrib's [new-component guidance](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/docs/new-components.md) expects a component to be built, hosted, used, and supported externally before donation, with a cross-company sponsor and **at least three codeowners** (including an approver/maintainer). External hosting is therefore the **main path**, not a fallback:

1. Ship a versioned external Go module + custom OCB distribution (binary + container image) from our own repo — intended GoogleCloudPlatform placement **subject to OSS release approval** (§6) — Apache-2.0, CI at contrib standards, first public artifact gated on the §11.1 baseline.
2. List in the OTel registry **at M2 with Alpha status** (so design partners can discover and install it; entry updated to Beta at M4); recruit named pilot users.
3. Publish schema contract, reliability (fault-injection results), and reproducible performance evidence.
4. Secure ≥ 3 codeowners and a sponsor from another company.
5. Submit the donation issue and the prescribed skeleton → implementation → Alpha/distribution PR sequence.

Prior-art history, corrected: #34809 closed after inactivity; PR #46280 was the implementation that stalled at the sponsor gate. Any codeowner-composition claims beyond the three-codeowner rule (e.g., vendor-employee requirements) must be cited from current contrib docs before being stated as requirements.

### 11.1 Release-trust baseline (gates the first public artifact)

The distributed collector handles Google credentials and raw telemetry, so the first public binary/container requires more than normal CI:

- Protected-tag builds using short-lived OIDC publishing credentials (no long-lived registry tokens).
- Immutable digests; signed checksums and container signatures.
- The signed/checksummed release bundle includes the **released DDL, Terraform, IAM definitions, dedup SQL, and extraction UDFs** — not only binaries and images (users execute these against production datasets; they deserve the same integrity guarantees).
- Build provenance attestation and an SPDX or CycloneDX SBOM per release.
- Dependency, vulnerability, license, and secret scanning in CI and on release.
- A pinned, minimal OCB manifest (only the components the distribution actually ships).
- Non-root minimal container image, read-only root filesystem, no shell or debug endpoints by default.
- `SECURITY.md` with a private reporting channel, supported-version policy, remediation targets, and an artifact-revocation procedure.

### 11.2 Live-GCP CI credential-isolation contract

M1 requires live-BigQuery e2e tests and M2 adds public release automation, so how CI reaches GCP is itself a contract:

- Short-lived **workload identity federation** into a **dedicated, quota-limited test project** — no long-lived service-account keys in CI. **M1 (internal, repo/provider not yet approved): provider-neutral WIF from the approved internal CI environment.** M2 external release: GitHub Actions OIDC *if* the approved repository is hosted on GitHub. All other requirements (dedicated project, protected environments, fork isolation, quotas, cleanup, audit) apply in both phases.
- Per-run datasets with automatic cleanup and audit trails.
- **No cloud credentials for fork PRs or untrusted branches**; credentialed tests run only from trusted workflows / protected environments.

### 11.3 Runtime-security profile (gates the M2 external Alpha)

§11.1 covers the supply chain; the running artifact handles Google credentials and raw telemetry, so a minimal runtime profile also gates the first public release (fuller security guide remains M3):

- Least-privilege IAM documentation and first-append permission diagnostics (§9).
- Loopback/private OTLP receiver defaults in shipped configs, with TLS/authentication guidance for any network exposure.
- Redaction recipes and FR13 metadata-only diagnostics.
- **Encrypted persistent storage as the production requirement** for sensitive telemetry queues — or an explicit, documented risk-acceptance statement.
- A canary test that forces telemetry through disk-backed queue persistence and checks temporary files, backups, and cleanup paths — not only BigQuery and exporter logs. The marker is injected independently as a value and as an attribute/`kvlist` key so FR13's diagnostic-path rule is exercised.
- Installation examples pinned by digest, with signer and provenance verification instructions.

## 12. Alternatives considered

| Alternative | Assessment |
|---|---|
| Adopt/extend the existing external exporter (PR #46280's continuation) | **Rejected (Q1 decision, §6):** we must own the OSS repo, and OSPO/legal determined reuse is unavailable (evidence in the controlled ledger) — independent build under the §6.1 protocol. Also carries a legacy v0 schema (untagged JSON, stringified bodies) incompatible with §7.3. Retained as behavioral prior art (public docs only) and credited in the contrib donation. |
| Cloud Logging sink → BigQuery via `googlecloudexporter` | Log-oriented schema, extra product hop; latency/cost claims to be measured and cited in the comparison doc, not asserted. |
| Pub/Sub exporter + Dataflow/BQ subscription | Viable, but two extra managed services and a pipeline to operate; poor fit for P1. |
| Legacy `insertAll` streaming inserts | Legacy alternative (not deprecated); Storage Write API chosen for its delivery semantics (offset-free default stream with acknowledgement) and documented throughput; cost comparison is a dated benchmark under NFR7, not a categorical claim. |
| fluent-bit / Vector BigQuery outputs | Logs only, no trace semantics, outside the OTel Collector ecosystem. |

## 13. Milestones

| Milestone | Scope | Exit criteria |
|---|---|---|
| M0 — design & decisions (**complete**) | Ownership decision (§6: independent build in our own repo); schema + delivery contracts | Decision recorded ✔; **field-level** trace/log schemas (§7) and at-least-once delivery contract (§8.4) approved **as the `v0alpha1` baseline** (§7.5 stability lifecycle) ✔ — the span clustering default remains a candidate pending M3 benchmarks (log no-clustering is final). External repo creation/publication stays gated on GCP OSS release approval (§6) |
| M1 — internal engineering preview (**no public artifacts**) | **Entry criterion: first §6.1 provenance sign-off recorded before the first implementation commit** (OSPO *engagement* is preparation, not the gate). **OSS release request submitted and tracked at M1 entry** (approval itself remains the M2 gate). Internal Go module + collector build; trace/log export to pre-created tables; **FR11a startup validation** (epoch/label, schema, location); **fingerprint generation + namespace tuple (§7.4) with their golden cases**; §11.2 CI credential isolation in place; design-partner recruitment begins (named recruitment owner per §14) | **First-commit dependency locks recorded** (exact Collector, pdata, OTLP proto, managedwriter, and Go versions — a deliverable, not an intention); golden pdata fixtures include every §7.6 case (invalid UTF-8 and one-sided raw trace context included); complete span/log dedup SQL executes against fixture tables; end columns are REQUIRED in golden DDL; namespace grammar/unset/startup-rejection and fleet-atomic migration documentation pass review; exact §8.6 generated metadata passes; destination-state retry policy has unit/fault seams; Q7 fingerprint-always-on benchmark decision is recorded **before public DDL freeze**; live-BigQuery e2e passes internally for both signals under §11.2 CI (provider-neutral WIF); **incompatible-table startup-rejection test passes (FR11a)**; OSS release request status + escalation path recorded; recruitment owner, channels, outreach log, and follow-up date recorded, with **≥ 2 identified design partners as the target rather than an M1 exit blocker** |
| M2 — **first external Alpha** (schema published as **`v0alpha1`**) | FR6 ack contract incl. crash-replay semantics, FR7 request sizing, §8.1 bounded appends + shutdown replay, FR5 queue/retry, FR12a delivery-critical telemetry, §8.4 duplicate-bound invariant, FR13 diagnostics; §11.1 release-trust baseline; **§11.3 runtime-security profile**; §9 M2 destination-governance minimum profile; OSS release approval obtained | FR6 tested as **two cases** on the pinned Collector: (i) in-process subset retry, (ii) crash replay with deterministic re-filtering; persistent-queue SIGTERM/crash-and-restart replay passes (or claim downgraded to best-effort), and a separate memory/no-queue shutdown test proves the documented non-durable outcome; destination schema drift, table deletion, and IAM revocation/repair tests prove valid queued rows are retried within the repair window rather than immediately dropped; dual-signal saturation proves aggregate in-flight bounds and **zero timeout-induced retries** at reference load; fault-injection asserts the ≤ N+1 invariant within its §8.4 scope; NFR3 outcome matrix validates the full 600s outage and publishes measured queue request count, bytes, and headroom (retry horizon > outage duration); **support-ownership package published**: named maintainer/support owner + triage cadence, supported Collector/pdata versions, known Alpha limitations, upgrade & schema-migration policy, artifact-withdrawal procedure; **provisional sizing/cost envelope published** (one reproducible workload: rows/s, p99 ack latency, memory, queue sizing, ingestion + dedup-query cost — marked provisional); external-Alpha exit gate: CUJ-1 executed from a clean GCP project using only the released artifact plus versioned DDL/Terraform, IAM setup, collector config, dedup SQL, extraction UDFs, and representative queries; OTel registry listing published with **Alpha** status |
| M3 | Optional auto-create (FR11b — startup validation moved to M1/FR11a), full security guide + CUJ-3 canary suite (extends the §11.3 M2 profile), **multi-tenant routing recipe (FR14/CUJ-4)**, extended self-telemetry (FR12b), span clustering benchmark | Chaos, security-canary, and reproducible performance tests pass; **CUJ-4 test proves a spoofed resource attribute cannot redirect data to another tenant's exporter**; NFR1/NFR2/NFR7 formal evidence published (supersedes the provisional M2 envelope); span clustering default frozen |
| M4 — external Beta adoption (**v1 schema freeze happens here**, per §7.5) | Pilot program with explicit, **task-based** thresholds | ≥ 3 independent organizations running ≥ 60 days each with retained usage; each pilot **independently completes CUJ-1 from the released artifact**, recording: time to first useful query, span/log extraction without undocumented schema knowledge, correct use of dedup SQL and UDFs, a reusable dbt model/dashboard (or equivalent), and ≥ 1 repeated use or successful upgrade; v1 schema frozen after ≥ 2 partners complete the §7.5 validation workflows; OTel registry entry updated from Alpha to Beta |
| M5 | Contrib donation | Cross-company sponsor, ≥ 3 codeowners, prescribed contrib PR sequence completed |

Beta maturity is defined by adoption, compatibility, reliability, and performance of **traces/logs**. Metrics and exactly-once are independent future proposals, not Beta prerequisites.

## 14. Risks & mitigations

- **OSS release approval — two distinct outcomes, not one.** (a) *GoogleCloudPlatform placement denied but release approved:* use another **OSPO-approved Google-managed** organization. (b) *External OSS release denied:* M2–M5 are **blocked** — Google's process requires approval before publication, and an alternative organization is not a workaround. Internal M1 work (post provenance sign-off) proceeds in either case.
- **Community splitting (second generic exporter).** Resolved posture: courtesy outreach to the prior implementation's maintainer (§6), prior-art credit in the donation proposal, open invitation to contribute to our repo. The §6.1 independent-implementation protocol (ledger, similarity review, OSPO sign-off) exists precisely so provenance questions can't stall the donation.
- **No contrib sponsor (third attempt).** External-first path means the component is useful regardless; donation proceeds only with adoption evidence and three codeowners in hand.
- **Schema contract churn.** Versioned schema + golden fixtures + additive-only evolution rules; fixtures run in CI on every change.
- **Duplicate-data surprises for analysts.** At-least-once stated everywhere (§8.4); dedup SQL shipped and tested; outage CUJ validates bounds.
- **Sensitive data leakage via logs or queue.** FR13 (metadata-only diagnostics), canary-secret CI test, persistent-queue security guidance.
- **Quota/cost surprises (Storage Write API throughput pricing).** Pricing model, per-table/project throughput quotas, and batching guidance documented; `sent_bytes` metric exposed.
- **Design partners / pilots fail to materialize.** A named recruitment owner is assigned at M1 entry with channels (existing SDK/BQAA users, OTel community, internal GCP workloads), an outreach log, and a deadline aligned to the M2 date. Identifying ≥ 2 partners is an M1 target, not an exit blocker: if < 2 partners by the M2 gate, Alpha may still ship, but the schema **stays `v0alpha`** and v1 does not freeze; if M4's ≥ 3-org threshold isn't met, Beta is deferred — **v1 is never frozen without the promised validation unless an explicit exception is approved and recorded on this issue**.

## 15. Open questions

1. ~~Adopt/collaborate/fork/build~~ — **Resolved**: independent **build in our own OSS repo** (§6); ownership requirement plus the OSPO/legal reuse determination (evidence in the controlled ledger) close off adoption/forking. Repository name pending; intended GoogleCloudPlatform placement **subject to OSS release approval**. Non-blocking: courtesy outreach per §6. Pre-implementation gate: §6.1 provenance sign-off.
2. ~~Final `AnyValue` tag grammar~~ — **Resolved** ([decision](https://github.com/GoogleCloudPlatform/BigQuery-Agent-Analytics-SDK/issues/364#issuecomment-4978169336)): `otlp-protojson-v1` (§7.3); the prior exporter's encoding treated as legacy v0, not reproduced.
3. Span clustering default `service_name, name` — confirm with M3 benchmarks before freezing the contract.
4. ~~Log `record_fingerprint` hash inputs~~ — **Resolved** (same decision, since superseded in detail by §7.4): resource attributes included; exact identity is the typed (`source_record_namespace`, `source_record_id`) tuple; when exact identity is unavailable the complete three-expression query uses the namespace-scoped (`COALESCE(source_record_namespace, '')`, `record_fingerprint`) content key, falling back table-wide when no namespace is configured. The pre-JCS `fp1` root grammar is now explicit rather than left to implementation choice.
5. Should the reference workload definition live in this repo or the exporter repo? (Proposal: exporter repo, linked from here.)
6. ~~Column modes for `end_timestamp` / `end_time_unix_nano` / `duration_ns`~~ — **Resolved:** all three are REQUIRED because §7.6 rejects spans without a valid start and end; §7.1 and the complete span dedup query now agree.
7. `record_fingerprint` is computed for every log even when trusted exact identity exists — **M1-owned decision:** benchmark its CPU/storage cost and record keep/drop outcome before public DDL/golden-schema freeze; M1 may not silently carry the question into M2.

## 16. References

- Contrib exporter directory: https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/exporter
- New component process: https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/docs/new-components.md
- Prior request: https://github.com/open-telemetry/opentelemetry-collector-contrib/issues/34809
- Prior implementation PR (description only, per §6.1): https://github.com/open-telemetry/opentelemetry-collector-contrib/pull/46280
- BigQuery Storage Write API: https://cloud.google.com/bigquery/docs/write-api
- Google OSS release process: https://opensource.google/documentation/reference/releasing (and approval guidance: https://opensource.google/documentation/reference/releasing/approval)
- Collector partial-error semantics: https://pkg.go.dev/go.opentelemetry.io/collector/consumer/consumererror
- Collector exporterhelper queue/retry/timeout semantics: https://pkg.go.dev/go.opentelemetry.io/collector/exporter/exporterhelper
- Collector batch processor semantics and defaults: https://pkg.go.dev/go.opentelemetry.io/collector/processor/batchprocessor
- Collector exporterhelper metric metadata: https://github.com/open-telemetry/opentelemetry-collector/blob/main/exporter/exporterhelper/metadata.yaml
- pdata timestamp and fixed-width ID types: https://pkg.go.dev/go.opentelemetry.io/collector/pdata/pcommon
- BigQuery `TIMESTAMP` domain: https://cloud.google.com/bigquery/docs/reference/standard-sql/data-types#timestamp_type
- Go JSON invalid-UTF-8 behavior: https://pkg.go.dev/encoding/json#Marshal
- RFC 8785 JSON Canonicalization Scheme: https://www.rfc-editor.org/rfc/rfc8785.html
- Design review driving this revision: https://github.com/GoogleCloudPlatform/BigQuery-Agent-Analytics-SDK/issues/364#issuecomment-4978066434

Start with the §6.1 provenance sign-off and the pinned Collector semantics before examining the proposed exporterhelper, managedwriter, metadata.yaml, and custom OCB entry points. No implementation should begin until the sign-off is recorded; the design is complete when its stated contracts, tests, support package, and OSS release gates are satisfied.

Written by the indexing model from the issue text.