Implement CMEK Encryption Support for TiCDC Next-Gen Architecture

Background

TiDB already supports Customer-Managed Encryption Keys (CMEK) encryption in the Next-Gen architecture. TiCDC needs to implement similar encryption capabilities to secure data before flushing it to disk during change data capture operations.

Objectives

Implement CMEK encryption support for TiCDC Next-Gen with the following key components:

1. Keyspace Encryption Metadata Retrieval

   • Fetch keyspace encryption metadata from TiKV

2. Master Key Management

   • Retrieve master keys from the Key Management System (KMS)

3. Data Encryption/Decryption

   • Encrypt data using data-keys (EncryptionKey) before writing to disk
   • Decrypt data using the same data-key when reading from disk

Design Overview

The implementation should follow a similar approach to TiKV's CMEK encryption design:

• When a changefeed is created, TiCDC fetches data from TiKV and writes it to disk
• Before writing, encrypt the data using the data-key (EncryptionKey)
• When reading, decrypt the data using the same data-key
• Use EncryptionMetaManager to manage data-keys with caching support
• If no cached key exists, retrieve it from TiKV and cache for future use

Data Format Considerations

The implementation will add 4 bytes for encryption metadata, which may introduce backward incompatibility with previous implementations. This needs to be carefully handled to ensure smooth upgrades.

Key Features

• Support for key rotation
• Efficient key caching mechanism
• Backward compatibility handling for existing deployments