matrixorigin / matrixorigin/matrixone

[Bug]: concurrent CREATE INDEX can omit a successful UPDATE from the new index

Open
#28,931 7 comments 0 reactions 1 assignee Claimed by @aptend View on GitHub
kind/bug needs-triage
Dominant language
Go
Stars
1.9k
Forks
311
Avg merge
1d 3h
Merged PRs (30d)
768

Description

## Description

`CREATE INDEX` can finish successfully while omitting a concurrently committed `UPDATE` from the new index. The base row remains present, but plans using the new index return an incomplete result.

The same race is more severe for a unique index: an `UPDATE` that creates a duplicate value and `CREATE UNIQUE INDEX` can both return success. The base table then contains two equal values while the published unique index contains only one of them.

## Environment

- Branch: `main`
- Commit: `b347a62a30719cf3f2c8ca5a2da0dcfb11639f8c`
- Deployment: isolated local 1 LogService + 1 TN + 2 CN deployment. Each statement used a direct, independent CN SQL endpoint.
- Date: 2026-09-15

## Regular index reproduction

1. Create and seed:

```sql
CREATE TABLE t(id INT PRIMARY KEY, payload VARCHAR(40) NOT NULL);
INSERT INTO t
SELECT result, CONCAT('payload-', result)
FROM generate_series(1, 2000);
```

2. Submit through two CNs at the same time:

```sql
-- CN 1
UPDATE t SET payload = 'changed-payload' WHERE id BETWEEN 501 AND 1000;

-- CN 2
CREATE INDEX ix_payload ON t(payload);
```

3. Both commands return success. In an affected round:

```sql
SELECT COUNT(*), SUM(id)
FROM t IGNORE INDEX(ix_payload)
WHERE payload = 'changed-payload';
-- 500, 375250

SELECT COUNT(*), SUM(id)
FROM t FORCE INDEX(ix_payload)
WHERE payload = 'changed-payload';
-- 0, NULL
```

The mismatch remained after five seconds and was preserved by `CREATE SNAPSHOT`, `CLONE`, and `DATA BRANCH`: the source, clone, and branch each returned `500,375250` with `IGNORE INDEX` and `0,NULL` with `FORCE INDEX`.

## Unique index reproduction

1. Seed `t(id INT PRIMARY KEY, a INT NOT NULL)` with `id=a=1..2000`.
2. Submit concurrently through two CNs:

```sql
-- CN 1
UPDATE t SET a = 500 WHERE id = 501;

-- CN 2
CREATE UNIQUE INDEX uk_a ON t(a);
```

3. In affected rounds both commands return success. The table and index then disagree:

```sql
SELECT id, a FROM t IGNORE INDEX(uk_a) WHERE a = 500 ORDER BY id;
-- (500,500), (501,500)

SELECT id, a FROM t FORCE INDEX(uk_a) WHERE a = 500;
-- (500,500)
```

`SHOW CREATE TABLE` reports `UNIQUE KEY uk_a (a)`. Later `INSERT ... (3000,500)` and `UPDATE ... SET a=500` are rejected as duplicates even though the already committed pair remains in the base table. The inconsistency survived a full 1 LogService + 1 TN + 2 CN restart and was copied by snapshot clone and data branch.

## Expected behavior

An index built successfully with a concurrently successful update must contain that update. For the unique-index case, the operations must serialize or one operation must fail; MO must not publish a unique constraint when the base table contains duplicate keys.

## Controls

- Regular-index sequential `UPDATE` then `CREATE INDEX`, and the reverse order, each returned the expected rows in normal and forced-index queries.
- Generated-column `CREATE INDEX`, `ADD generated column`, and `MODIFY` source-column paths each completed 100 iterations with source values, generated values, and forced-index results matching.
- MySQL 8.0.45 ran the conflicting `UPDATE` plus `CREATE UNIQUE INDEX` race for 50 iterations: the update won and the unique-index build was rejected every time; it never produced a successful index over duplicate base rows.

## Evidence

Local redacted drivers and outputs are retained under `evidence/ddl_cross_complete/`:

- `ordinary_index_create_dml_race.sh`
- `ordinary_index_base_scan_audit_b347.out`
- `ordinary_index_corruption_lifecycle_b347.out`
- `add_unique_conflicting_update_race.sh`
- `add_unique_conflicting_update_preserved_queries_b347.out`
- `add_unique_corruption_after_restart_b347.out`
- `add_unique_corruption_lifecycle_b347.out`
- `mysql_create_unique_conflicting_update_race_8.0.45.out`

## Code analysis

`Scope.CreateIndex` locks table metadata and delegates regular and unique index construction through separate handlers before publishing the updated table constraint in `pkg/sql/compile/ddl.go`. The observed mismatch shows that the committed update and index build do not share a complete handoff for this interleaving. The exact missing synchronization point is not yet confirmed.

## Regression coverage

Do not baseline either incorrect result. After a product fix, add a dual-CN repeated race to MOTR: ordinary index build versus indexed-column update with `IGNORE INDEX`/`FORCE INDEX` equivalence, and unique index build versus a conflicting update with a postcondition that the base table never contains duplicate unique keys. The minimal fixture reproduces on 2,000 rows, so it does not need big-data or chaos coverage.

## Related

- #26804 is closed and covered a failed unique DDL after a longer indexed-UPDATE history that resurrected prior versions and duplicated primary keys. This report observes a successful `CREATE UNIQUE INDEX` that publishes an incomplete unique index over a current duplicate secondary key.
- #27487 concerned concurrent `INSERT` rows missing from a newly created index.
- #28639 concerns post-DDL index-name visibility on another CN.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.