dotCMS / dotCMS/core

--migrate: Testcontainers IT suite (base + edge cases)

Open
#37,241 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

dotCMS : Analytics Team : Falcon Type : Task
Dominant language
Java
Stars
970
Forks
486
Avg merge
3d 33m
Merged PRs (30d)
170

Description

Part of #37237. Depends on the Ledger & verification, Core migration engine, and CLI surface tickets.

Summary

Migration-specific integration tests (in dotCMS/dot-ca-event-manager), extending the existing Testcontainers harness (2 ClickHouse replicas + Keeper + Toxiproxy, per .claude/testing.md). May land as two PRs (base cases, then edge cases) even though it's tracked as one ticket.

Scope

Base cases
Seed via docker/init/*.sql, apply a candidate migration through the real runner, assert:

  • pre-existing queries are unaffected
  • new columns/defaults are correct
  • the schema change propagated to the second replica (poll system.columns, the same shape as the existing assertReplicatedCount() helper)
  • applying the same migration twice is a no-op

Edge cases — grounded in the real bug already caught once during design (mutations_sync=2 breaking retry-safety), not hypothetical:

  • a metadata-only ADD COLUMN case
  • a mutation-triggering case with mutation_id capture + poll to completion
  • checksum-mismatch aborts the run
  • V0-mismatch aborts the run
  • two concurrent runners contend for the Keeper lock and only one proceeds
  • a script interrupted mid-way is safe to re-run in full
  • a dropped connection during a long mutation (Toxiproxy) doesn't cause a retry to enqueue a duplicate mutation
  • a RENAME TABLE on a table with an attached refreshable MV, including one with a DEPENDS ON chain to another refreshable MV — three of the ten real environment-rollout runbooks (docs/migrations/) independently flag this as genuinely less battle-tested than a normal MV rename, not hypothetical caution

Verification (end-to-end, once this + all core tickets land)

Do a dry run of one real migration from the environment rollout (e.g. the simplest one, docs/migrations/07_engagement_daily.md) through the actual --migrate tool against a local stack, to confirm the tool works against genuine migration content, not just synthetic test SQL.

Repo

dotCMS/dot-ca-event-manager

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with .claude/testing.md and the existing Testcontainers harness in dotCMS/dot-ca-event-manager, then inspect docker/init/*.sql and the assertReplicatedCount() helper. Implement the base and edge-case integration coverage described, including the real migration dry run using docs/migrations/07_engagement_daily.md once dependencies land. Done means the listed failure, retry, replication, locking, and refreshable-MV cases are verified.

Written by the indexing model from the issue text.

Assessment

Tech stack
clickhouse, docker, java, sql
Domain
cli, databases, devops, testing-qa
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.