matrixorigin / matrixorigin/matrixone
[POC Use Case] Atomic-swap ETL pattern (build into _shadow → RENAME swap) fails on MatrixOne due to compounded DDL gaps
- Dominant language
- Go
- Stars
- 1.9k
- Forks
- 311
- Avg merge
- 1d 3h
- Merged PRs (30d)
- 768
Description
## Version
MatrixOne **v4.0.0-rc2** (`8.0.30-OmniFabric-v4.0.0-rc2`)
## Use Case
We are running an use case that follows a standard Medallion-architecture ETL pipeline on top of MatrixOne:
```
MongoDB sensor stream ──► Bronze (bronze_raw)
│
▼
Silver (analytics.silver_*) ── 11 tables
│
▼
Gold (analytics.gold_*) ── 3 tables
│
▼
Superset dashboard (live)
```
Silver+Gold rebuild runs every ~15 minutes. A naive `TRUNCATE + INSERT` against the live tables would leave the dashboard staring at empty tables for 10+ minutes per rebuild, so we adopted the **standard atomic-swap pattern** used widely with MySQL/MariaDB:
```sql
-- once at bootstrap
CREATE TABLE IF NOT EXISTS silver_X_shadow LIKE silver_X;
-- every rebuild
TRUNCATE silver_X_shadow;
INSERT INTO silver_X_shadow SELECT ...; -- builds new data into _shadow
-- ... repeat for all 14 tables ...
-- atomic swap: live serves old data until the moment of RENAME
RENAME TABLE silver_X TO silver_X_old,
silver_X_shadow TO silver_X,
silver_X_old TO silver_X_shadow;
```
On MySQL 8 / MariaDB this is rock-solid. On MatrixOne v4.0.0-rc2 it falls apart and leaves the database in inconsistent states that the loop cannot self-heal from.
## Observed Behavior
After running the loop for ~12 hours we end up with:
- `silver_intelie_readings_cleaned_shadow` — **missing** (a previous RENAME removed it half-way through a multi-pair statement)
- `gold_pump_kpis_shadow` — **missing** (same cause)
- Several `silver_*_old` tables — left behind from aborted swaps
- Live `gold_pump_kpis` stuck at the row count from the last successful rebuild
- Loop logs flooded with mixed errors:
- `ERROR 1050: table 'silver_X_shadow' already exists` (from `CREATE TABLE IF NOT EXISTS` — see sub-issue 1)
- `ERROR at line 1: ExpectedDup` (intermittent INSERT failures)
- `ERROR 1146: no such table 'gold_pump_kpis_shadow'` (after partial RENAME — see sub-issue 2)
- `Duplicate entry '(HP-101, 2026-01-15 00:00:00)' for key '(pump,datetime)'`
## Root Cause Family
We traced the loop instability to **three independent DDL behavior gaps** in MatrixOne. Filing them as separate sub-issues so each can be triaged independently. **Any single one of these is enough to break the atomic-swap pattern**; the three of them together make the pattern unusable.
1. `CREATE TABLE IF NOT EXISTS` is not a no-op when the table exists. → see sub-issue
2. `RENAME TABLE` with multiple pairs is not atomic; partial failures destroy tables. → see sub-issue
3. Multi-statement `-e` batches do not have consistent error-stop semantics. → see sub-issue
## Why This Matters
The atomic-swap pattern (build into shadow → RENAME swap) is one of the two canonical zero-downtime ETL rebuild patterns in the MySQL ecosystem (the other is transactional MERGE/UPSERT, which has its own MO issues — see matrixone#24408). Users migrating any non-trivial pipeline from MySQL / MariaDB / Aurora to MatrixOne will hit this on day one.
## Our Current Workaround
We have switched the loop to `TRUNCATE + INSERT` directly against the live tables, accepting a ~10-minute window per rebuild where the dashboard shows zero rows. This is not acceptable for production but allows the demo to continue.
## Reference
- Reproductions available on request
- Related issues already filed: matrixone#24406, matrixone#24407, matrixone#24408, matrixone#24409, matrixone#24410, matrixone#25115
- Sub-issues for individual gaps will be linked below as comments once they are filed.
Contributor guide
Assessment
This issue has not been assessed yet.