Differential transformer: migration output must match live ingestion
Nobody has claimed this yet.
- Dominant language
- TypeScript
- Stars
- 0
- Forks
- 0
- Avg merge
- 3m
- Merged PRs (30d)
- 1
Description
Problem
The migration tool's transform (src/transform/normalize.ts) and the live ingestion path in countly-platform (api/ingestor/requestProcessor.ts / event-sink normalization) are two independent implementations of the same conversion. After a customer is migrated, their historical data (produced by this tool) and their live data (produced by the platform) must be identical in shape for the same input — any drift means a customer's history disagrees with their new data, per key, silently.
Today nothing enforces this. A change to either side lands green and the divergence is only discovered in customer data.
Proposal — a two-sided contract test
This is a contract between two codebases, so drift can originate from either side; enforcement must exist on both:
- countly-platform CI gains a small job that generates golden fixtures:
(input doc) → (ClickHouse row produced by the live ingestion normalization). A platform PR that changes normalization produces a visible, reviewable diff in the goldens — the change is signed, not silent. - This repo's CI asserts
transformBatch()reproduces the goldens exactly.
Fixture corpus
- Harvested real-world oddities: dry-run report findings and DLQ contents from actual migrations (every value class that has actually occurred).
- Property-based hostile values: integers beyond Int64, NaN/Infinity, mixed types per key, deep nesting, hostile strings, missing/blank required fields.
Coercion policy alignment
The transform's rules must be the answer to "what would live ingestion do":
- Countly-owned fields (
c,s,dur, timestamps): clamp to column range. - Customer-owned keys (
sg.*,custom.*,cmp.*): never guess — values that don't fit numeric representation are stringified losslessly.
Every rule promoted here should carry a golden fixture proving both sides agree.
Acceptance
- A normalization change in either repo fails the other repo's CI (or produces a reviewed golden diff) — verified by deliberately introducing a divergence in a test branch.
- Corpus covers all currently-known rejection classes (incl. the empirically verified ones: oversized ints, unparseable DateTime — CH error codes 41/117 family).
Corresponds to item D4 of the internal drill-migration overhaul plan (companion items: D1 sampled dry run, D2 coercion policy, B2 DLQ).
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reading src/transform/normalize.ts and the referenced countly-platform paths: api/ingestor/requestProcessor.ts and event-sink normalization. Review the existing migration CI and determine how golden fixtures can be generated and consumed across both repositories. Done means known rejection classes and hostile values are covered, transformBatch() matches the goldens, and deliberate divergence is detected or produces a reviewed diff.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- clickhouse, typescript
- Domain
- backend, ci-cd, databases, testing
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100