--migrate: Ledger & verification (schema_migrations table, checksum, V0 baseline)
Nobody has claimed this yet.
- Dominant language
- Java
- Stars
- 970
- Forks
- 486
- Avg merge
- 3d 33m
- Merged PRs (30d)
- 170
Description
Part of #37237.
Summary
The schema_migrations table (in dotCMS/dot-ca-event-manager) and everything that reads/writes trust into it: the table itself, checksum verification, and the V0 baseline check. This is the first ticket in implementation order — the core apply loop (next ticket) and the CLI surface both depend on this existing.
Scope
1. analytics.schema_migrations ledger table
Add its CREATE TABLE to the docker/init/*.sql baseline: ReplicatedReplacingMergeTree(applied_at), ORDER BY (version), columns version, description, checksum, applied_at, applied_by, duration_ms. All reads for a given version must use FINAL — a plain ReplicatedMergeTree gives no per-version uniqueness guarantee, and the checksum-mismatch check below assumes exactly one row per version.
2. Checksum verification
SHA-256 of each migration file's raw bytes (not normalized for whitespace/line endings). Compare against the ledger for already-applied versions. On mismatch: abort the whole run with a clear error — never auto-overwrite the record, never warn-and-continue.
3. V0 baseline verification
On first run against a given cluster, diff live system.columns/system.tables against what docker/init/*.sql would produce. Record a V0 ledger row only on a clean match. Refuse to apply V1+ on any structural mismatch until reconciled by hand.
One specific, named exception, not a silent allowlist: a REFRESH EVERY interval difference on a refreshable MV is expected (dev/test uses a fast cadence for iteration speed, production intentionally runs slower for cost/load) and doesn't block V1+. The tool still prints an explicit warning naming the table and both values whenever it finds this specific difference — in both a real run and --dry-run (see CLI surface ticket) — so it stays visible rather than silently dismissed. Every other kind of difference (missing column, wrong type, wrong engine, wrong ORDER BY/PARTITION BY) still hard-aborts exactly as before.
Why this matters (grounding)
ClickHouse schema has already changed at least once outside any tracked mechanism (the experiment columns added for #37016) — different clusters may have reached today's schema via different, untracked paths. An unverified V0 would make the whole ledger rest on an unverified premise. The REFRESH EVERY exception exists because docs/cross-environment-analytics-viewing-plan.md (in dot-ca-event-manager) confirms dev uses REFRESH EVERY 30 SECOND while production intentionally uses REFRESH EVERY 15 MINUTE+ — a real, expected divergence, not drift.
Out of scope here
- The Keeper-based run lock and the actual apply loop (next ticket).
- The
--dry-run/--yesCLI flags themselves (CLI surface ticket) — this ticket just needs to make the underlying checks callable.
Repo
dotCMS/dot-ca-event-manager
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start in dotCMS/dot-ca-event-manager by reading docker/init/*.sql and the migration tooling that will consume the ledger. Inspect the existing schema definitions and the documented dev/production REFRESH EVERY values in docs/cross-environment-analytics-viewing-plan.md. Done means the ledger, raw-byte checksum checks, V0 structural verification, and explicit refresh-interval warning are callable for the later apply loop and CLI.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- sql
- Domain
- databases, tooling
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100