pingcap / pingcap/tidb

Implement end to end checksum for TiDB and TiCDC

Open
#42,747 0 comments 0 reactions 2 assignees Claimed by @zyguan View on GitHub
sig/transaction type/new-feature
Dominant language
Go
Stars
40.5k
Forks
6.2k
PR merge metrics
PR metrics pending

Description

## Background

TiCDC is an important component for TiDB to synchronize data to various downstream systems. When synchronizing data to downstream systems, data integrity is especially important. However, TiCDC does not support end-to-end data integrity verification yet.

## Spec

Provide below cluster level boolean type option in TiDB side.
```
tidb_enable_row_level_checksum = [true|false] # the default value is false.
SET GLOBAL tidb_enable_row_level_checksum = true;
```

After the customer enables this option, every data change for a row in non-system databases will append an invisible field that is used to store a computed checksum value based on the content of the row. This invisible field is just for data correctness checking purposes and is transparent to the customer.

TiCDC and end users would use this checksum value to verify the data integrity.

### Development tracking for the TiDB part

- [x] Add a checksum function `tidb_row_checksum` to return the checksum value of a row. #43479
- [x] Add checksum-related utilities in tidb, in this case, CRC32 would be used, the calculation method is shown as the following. #42859 #43141
- [x] Let tidb be aware of the origin state (none or public) of a column if its current state is not public. -- we always append two checksums if there is a column whose state is not public, thus no need to know the direction of state transform.
- [x] Support writing rows with checksum values #43163
- [x] Add a global system variable `tidb_enable_row_level_checksum` to enable or disable the checksum calculation when inserting new rows. When it's enabled, multi-schema change will be blocked.
- [x] Make it work with the DDL `add column` schema change, and generate two checksum values if necessary.
- [x] Make it work with the DDL `drop column` schema change, and generate two checksum values if necessary.
- [x] Make it work with the DDL `modify column` schema change, and generate two checksum values if necessary.
- [x] Calculate the row checksum in the `tablecodec` package when `EncodeRow` function is used. Calculate the CRC32 result for each column when executing `encodeRowCols`, a checksum result is returned finally.
- [x] Append the checksum header and checksum result information to the encoded row according to the extended row format protocol.
- [ ] Keep the read request processing compatibility.
- [x] Skip the checksum part processing for `chunckDecoder` in tidb if necessary. #42859
- [x] Skip the checksum part processing in `internal_handle_request` and `PointGetter` for chunk encoding processing in tikv if necessary. https://github.com/tikv/tikv/pull/14611
- [x] Skip the checksum part processing in tiflash if necessary. -- tiflash decodes a row value by [`appendRowV2ToBlockImpl`](https://github.com/pingcap/tiflash/blob/df704b8ca6f98754f871b307130f93d2604ca55e/dbms/src/Storages/Transaction/RowCodec.cpp#L390). it iterates columns and decodes them one by one [here](https://github.com/pingcap/tiflash/blob/df704b8ca6f98754f871b307130f93d2604ca55e/dbms/src/Storages/Transaction/RowCodec.cpp#L499), that is, the checksum part shall be already discarded in the current implementation.
- [ ] Skip the checksum part processing in tikv client libs if necessary. https://github.com/tikv/client-java/pull/739
- [ ] Add telementry for the new feature.
- [ ] Compatibility tests, the checksum extended part should not impact the backward compatibility, and downgrade is supported when the checksum row format is used.

### Development tracking for the TiKV part
- https://github.com/tikv/tikv/issues/14528

### Development tracking for the TiCDC part
- https://github.com/pingcap/tiflow/issues/8718

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.