matrixorigin / matrixorigin/matrixone

[POC Use Case] Atomic-swap ETL pattern (build into _shadow → RENAME swap) fails on MatrixOne due to compounded DDL gaps

Open
#25,118 1 comment 0 reactions 1 assignee Claimed by @LeftHandCold View on GitHub
severity/s0
Dominant language
Go
Stars
1.9k
Forks
311
Avg merge
1d 3h
Merged PRs (30d)
768

Description

## Version

MatrixOne **v4.0.0-rc2** (`8.0.30-OmniFabric-v4.0.0-rc2`)

## Use Case

We are running an use case that follows a standard Medallion-architecture ETL pipeline on top of MatrixOne:

```
MongoDB sensor stream ──► Bronze (bronze_raw)


Silver (analytics.silver_*) ── 11 tables


Gold (analytics.gold_*) ── 3 tables


Superset dashboard (live)
```

Silver+Gold rebuild runs every ~15 minutes. A naive `TRUNCATE + INSERT` against the live tables would leave the dashboard staring at empty tables for 10+ minutes per rebuild, so we adopted the **standard atomic-swap pattern** used widely with MySQL/MariaDB:

```sql
-- once at bootstrap
CREATE TABLE IF NOT EXISTS silver_X_shadow LIKE silver_X;

-- every rebuild
TRUNCATE silver_X_shadow;
INSERT INTO silver_X_shadow SELECT ...; -- builds new data into _shadow
-- ... repeat for all 14 tables ...

-- atomic swap: live serves old data until the moment of RENAME
RENAME TABLE silver_X TO silver_X_old,
silver_X_shadow TO silver_X,
silver_X_old TO silver_X_shadow;
```

On MySQL 8 / MariaDB this is rock-solid. On MatrixOne v4.0.0-rc2 it falls apart and leaves the database in inconsistent states that the loop cannot self-heal from.

## Observed Behavior

After running the loop for ~12 hours we end up with:

- `silver_intelie_readings_cleaned_shadow` — **missing** (a previous RENAME removed it half-way through a multi-pair statement)
- `gold_pump_kpis_shadow` — **missing** (same cause)
- Several `silver_*_old` tables — left behind from aborted swaps
- Live `gold_pump_kpis` stuck at the row count from the last successful rebuild
- Loop logs flooded with mixed errors:
- `ERROR 1050: table 'silver_X_shadow' already exists` (from `CREATE TABLE IF NOT EXISTS` — see sub-issue 1)
- `ERROR at line 1: ExpectedDup` (intermittent INSERT failures)
- `ERROR 1146: no such table 'gold_pump_kpis_shadow'` (after partial RENAME — see sub-issue 2)
- `Duplicate entry '(HP-101, 2026-01-15 00:00:00)' for key '(pump,datetime)'`

## Root Cause Family

We traced the loop instability to **three independent DDL behavior gaps** in MatrixOne. Filing them as separate sub-issues so each can be triaged independently. **Any single one of these is enough to break the atomic-swap pattern**; the three of them together make the pattern unusable.

1. `CREATE TABLE IF NOT EXISTS` is not a no-op when the table exists. → see sub-issue
2. `RENAME TABLE` with multiple pairs is not atomic; partial failures destroy tables. → see sub-issue
3. Multi-statement `-e` batches do not have consistent error-stop semantics. → see sub-issue

## Why This Matters

The atomic-swap pattern (build into shadow → RENAME swap) is one of the two canonical zero-downtime ETL rebuild patterns in the MySQL ecosystem (the other is transactional MERGE/UPSERT, which has its own MO issues — see matrixone#24408). Users migrating any non-trivial pipeline from MySQL / MariaDB / Aurora to MatrixOne will hit this on day one.

## Our Current Workaround

We have switched the loop to `TRUNCATE + INSERT` directly against the live tables, accepting a ~10-minute window per rebuild where the dashboard shows zero rows. This is not acceptable for production but allows the demo to continue.

## Reference

- Reproductions available on request
- Related issues already filed: matrixone#24406, matrixone#24407, matrixone#24408, matrixone#24409, matrixone#24410, matrixone#25115
- Sub-issues for individual gaps will be linked below as comments once they are filed.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.