matrixorigin / matrixorigin/matrixone

CDC TPCC 一致性校验失败: bmsql_order_line 同步滞后导致 oorder/order_line 数据不一致

Open
#26,949 8 comments 0 reactions 1 assignee Claimed by @jiangxinmeng1 View on GitHub
kind/bug severity/s-1
Dominant language
Go
Stars
1.9k
Forks
311
Avg merge
1d 3h
Merged PRs (30d)
768

Description

## 问题描述

08-10 nightly 129 单机回归中,MO CDC MO TEST 的步骤 **1.3 downstream mo data tpcc consistency verification** 失败。

## 失败详情

CDC TPCC 数据一致性校验 SQL 返回非空结果(应当为空):

```sql
(select o_w_id, o_d_id, sum(o_ol_cnt) from bmsql_oorder group by o_w_id, o_d_id)
except
(select ol_w_id, ol_d_id, count(ol_o_id) from bmsql_order_line group by ol_w_id, ol_d_id);
```

下游 MO 中 `bmsql_oorder` 和 `bmsql_order_line` 数据不一致,返回了大量 warehouse+district 维度的差异行。

## CDC Checkpoint 分析

校验时 CDC task 各表的 checkpoint 时间戳:

| 表 | Checkpoint | 延迟 |
|---|---|---|
| `bmsql_config` | 05:28:17 | 正常 |
| `bmsql_customer` | 05:28:17 | 正常 |
| `bmsql_district` | 05:28:17 | 正常 |
| `bmsql_history` | 05:28:17 | 正常 |
| `bmsql_item` | 05:28:17 | 正常 |
| `bmsql_new_order` | 05:28:17 | 正常 |
| `bmsql_oorder` | 05:28:17 | 正常 |
| `bmsql_warehouse` | 05:28:17 | 正常 |
| **`bmsql_order_line`** | **05:20:01** | **滞后 ~8 分钟** |
| **`bmsql_stock`** | **05:19:33** | **滞后 ~8.5 分钟** |

`bmsql_order_line` 和 `bmsql_stock` 两张大表的 CDC 同步严重滞后于其他表(约 8 分钟),导致一致性校验时 oorder 已同步但 order_line 尚未同步完成。

## 环境信息

- **分支:** main
- **Commit:** `00cb7ef3169a6c61f358774ac5c690a405ac162d`
- **Runner:** mo-srv-129(单机)
- **CI Run:** https://github.com/matrixorigin/mo-auto-test/actions/runs/31405749817/job/93598905378
- **测试场景:** TPCC 10 warehouses / 100 terminals 并发写入期间,CDC 同步到下游 MO,然后做一致性校验

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.