[Bug] CDC streaming job with offset=latest/earliest fails on SSL-required MySQL because SSL properties are applied after JDBC connection
- Dominant language
- Java
- Stars
- 15.9k
- Forks
- 3.9k
- Avg merge
- 2d 23h
- Merged PRs (30d)
- 520
Description
## Version and Environment
- **Apache Doris version**: 4.1.2 (also reproduced on 4.1.3-rc02, bug is present on master/4.2-SNAPSHOT)
- **Deployment mode**: Compute-storage-decoupled (cloud mode), but the bug also applies to shared-nothing mode
- **Source database**: MySQL 8.0 / Amazon Aurora MySQL 8.0 (`8.0.mysql_aurora.3.10.3`) with `require_secure_transport=ON`
## What happened
A CDC streaming job configured with `offset='latest'` (or `earliest`) and `ssl_mode='require'` immediately fails with a JDBC connection error at job startup. The job never begins replication.
## What was expected
The job should connect using SSL to resolve the current binlog position and then begin streaming.
## Minimal Reproduction
1. Configure MySQL/Aurora with `require_secure_transport=ON`:
```sql
-- On MySQL:
SET GLOBAL require_secure_transport = ON;
```
2. Upload a CA certificate to Doris (if using verify-ca, otherwise ssl_mode=require suffices):
```sql
CREATE FILE "mysql_ca.pem"
PROPERTIES ("url" = "file:///path/to/rds-combined-ca-bundle.pem", "catalog" = "internal");
```
3. Create a CDC streaming job:
```sql
CREATE JOB my_cdc_job
PROPERTIES (
'type' = 'insert',
'format' = 'cdc'
)
FROM MYSQL (
'host' = '10.0.1.100',
'port' = '3306',
'user' = 'cdc_user',
'password' = '***',
'database' = 'mydb',
'table' = 'orders',
'offset' = 'latest',
'ssl_mode' = 'require'
)
INTO TABLE mydb.orders;
```
4. The job fails immediately. The BE `cdc_client` log shows:
```
java.sql.SQLException: Could not create connection to database server.
Caused by: javax.net.ssl.SSLHandshakeException: No appropriate protocol
```
or (depending on MySQL configuration):
```
java.sql.SQLException: Connections using insecure transport are prohibited while --require_secure_transport=ON
```
## Root Cause
In `MySqlSourceReader.generateMySqlConfig()` (`fs_brokers/cdc_client/src/main/java/org/apache/doris/cdcclient/source/reader/mysql/MySqlSourceReader.java`), the startup mode switch block (starting at line 907 in tag 4.1.2) calls `initializeEffectiveOffset()` for `latest`, `earliest`, and timestamp modes. This method creates a `MySqlConnection` via `DebeziumUtils.createMySqlConnection(config)` — i.e., it opens a JDBC connection to the source database to resolve the current binlog file and position.
However, the SSL properties (`ssl_mode` → `database.ssl.mode` / `sslMode`, and `ssl_rootcert` → truststore path) are applied to `jdbcProperties` and `dbzProps` **after** the startup mode block (line 955+ in tag 4.1.2). So when `initializeEffectiveOffset()` builds its config and opens a connection, SSL is not yet configured. Against a MySQL instance with `require_secure_transport=ON`, the plaintext connection attempt is rejected.
## Affected Modes
- `offset='latest'` — always calls `initializeEffectiveOffset()`
- `offset='earliest'` — always calls `initializeEffectiveOffset()`
- Timestamp offset (13-digit epoch) — always calls `initializeEffectiveOffset()`
The `offset='initial'` and `offset='snapshot'` modes are NOT affected because they do not resolve binlog position at configuration time.
## Suggested Fix
Move the JDBC properties + SSL configuration block to before the startup mode switch block. When SSL properties are absent, the behaviour is identical (the JDBC connection simply doesn't use SSL). The reordering has no effect on non-SSL users.
Contributor guide
Research direction
Start in fs_brokers/cdc_client/src/main/java/org/apache/doris/cdcclient/source/reader/mysql/MySqlSourceReader.java, reading generateMySqlConfig(), initializeEffectiveOffset(), and the startup mode and JDBC/SSL configuration blocks. Verify the CDC job with latest, earliest, or timestamp offsets against SSL-required MySQL; done means the SSL properties are applied before binlog-position resolution while initial and snapshot behavior remains unchanged.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, mysql
- Domain
- databases, stream-processing
- Issue type
- Bug
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Quiet
- Clarity
- Clearly specified
- Newbie friendliness
- 76/100