ADORSYS-GIS / ADORSYS-GIS/lightbridge-authz

[Story]: Cutover: migrate the governance store's rows in, verify counts, decommission

Open
#588 0 comments 0 reactions 0 assignees View on GitHub
user-story
Dominant language
Rust
Stars
0
Forks
1
Avg merge
7h 7m
Merged PRs (30d)
237

Description

Parent epic: #581

### Story Statement

As the platform owner, every row the governance store ever collected (Copilot dailies, seat snapshots, executions/model_calls/tool_calls) lives in the usage store with asserted-equal counts, and the governance store's telemetry tables are dropped.

### Real Intent

Execute the cutover ADR-0027/governance ADR-0014 mandate: migrate, verify by counts, decommission. Hard cutover — no dual-write period, no dormant tables.

### Background and Context

Copilot data has a replay path already: governance-ctl archives raw NDJSON to S3 (prefix `copilot-governance/raw/`) and has a `replay` subcommand — replaying through the NEW day-grain ingest API exercises the true write path (the same principle as #528). Execution-grain rows have no raw archive and migrate directly, table-to-table. governance#167's no-loss bar (counts asserted, idempotent reprocessing) applies to both.

### Source of truth (links)

ADR-0027 Decision 1 / Consequences · governance ADR-0014 Decision 3 · governance#167 · governance#181 (the failing verify job — fix or retire as part of cutover)

### Acceptance Criteria

1. Copilot dailies + seats land via S3 replay through the authenticated day-grain ingest path; per-`(day, report)` counts match the governance store's `ingest_manifests` records; mismatches block the cutover loudly.
2. Execution/model-call/tool-call rows migrated with per-table count assertions; `(trace_id, span_id)` dedup means re-running the migration changes nothing.
3. Identity references resolve through `usage_identities` (provider user ids preserved verbatim; no shape validation).
4. After verification, the governance telemetry tables are dropped in the same coordinated change (governance-side PR) — not left dormant; `governance-ctl` is already writing to the usage API by then (governance-side story).
5. Dashboards/readers that pointed at the governance Postgres are repointed or retired in the same window (ai-helm#879/#880 successors noted, not silently broken).

### Out of Scope

Building the ingest paths themselves (prior stories); Grafana dashboard redesign.

### Dependencies and Blockers

Day/seat + execution grain stories; authenticated ingest; governance-side story "governance-ctl writes to the usage ingest API". This story lands LAST in the epic.

### Assumptions

The S3 raw archive is complete for the collected window (verify against `ingest_manifests` before trusting; governance#181's failing verify job is a red flag to resolve first).

### Implementation Notes

Sequence: freeze governance-ctl writes → replay/migrate → count assertions → cut governance-ctl over to the usage API → drop old tables. Keep the freeze window short; day-grain data is daily, so a same-day window is available.

### Test Expectations

Count-assertion harness output committed as evidence; a deliberately-corrupted-row rehearsal proving the mismatch path blocks (sabotage-first).

### Verification evidence

To be attached on the implementing PR: per-table before/after counts, replay logs.

### Human accountable owner

Stephane Segning Lambou

### AI Usage
AI-assisted (Claude): research and drafting. Reviewed and owned by Stephane Segning Lambou.

### Design addendum (2026-08-31): the historical backfill exception, pinned

Going forward, the replay/archive story is the collector's raw OTLP object-storage leg (#589), and per-tool archive code retires. **The one hard exception is this story's historical Copilot data: it never transited the collector, so neither the Grafana stack nor the new archive ever saw it.** governance-ctl's existing S3 NDJSON archive + `replay` is the ONLY source for that history and is used here exactly once, as the migration input (AC 1). After this cutover completes and counts are asserted, governance-ctl's bespoke archive path retires with the rest of its direct-Postgres sink — retirement is sequenced **after** this story, never before.

Contributor guide

Open the contributing guide

Research direction

Start with ADR-0027, governance ADR-0014, governance#181, and governance-ctl's S3 raw NDJSON replay path. Verify archive completeness against ingest_manifests, then freeze writes, replay or migrate the data, and run per-(day, report) and per-table count assertions, including an idempotent rerun and corrupted-row rehearsal. Done means counts match, identities resolve, readers are repointed or retired, governance-ctl uses the usage API, and the old telemetry tables are dropped.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, grafana, postgresql, rust
Domain
backend, cloud, databases, observability
Issue type
Refactor
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
32/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.