apache / apache/doris

[Feature] MySQL CDC: support regex (or wildcard) in exclude_tables / include_tables

Open
#67,484 0 comments 0 reactions 0 assignees View on GitHub
kind/feature
Dominant language
Java
Stars
15.9k
Forks
3.9k
Avg merge
2d 23h
Merged PRs (30d)
520

Description

### Search before asking

- [x] I had searched in the [issues](https://github.com/apache/doris/issues?q=is%3Aissue) and found no similar issues.

### Description

I would like to request adding regular expression (or wildcard) support for the `exclude_tables` and `include_tables` configurations in `CREATE JOB ... ON STREAMING FROM MYSQL`.

Currently, `exclude_tables` / `include_tables` perform exact string matching only:
`fe/fe-core/src/main/java/org/apache/doris/job/util/StreamingJobUtils.java`

```java
excludeTablesList = Arrays.asList(excludeTables.split(",")); // L378-380
// ...
if (includeTablesList.isEmpty() && !excludeTablesList.isEmpty() && excludeTablesList.contains(table)) { // L401-402
continue;
}
```

### Solution

Support regex (or at minimum glob-style wildcards) in both `exclude_tables` and
`include_tables`. Flink CDC's `tables` option already accepts regex
(e.g. `app_db.\.*`), so this would also align the two code paths.

Suggested semantics:

- Keep backward compatibility: a pattern with no regex metacharacters behaves exactly as today.
- Match against the bare table name (as today), not the qualified name, to avoid ambiguity.
- Document whether the match is full-match or partial-match (`matches()` vs `find()`) —
full-match is less surprising.

Two related nits found while investigating, worth fixing in the same change:

1. **No trimming.** `excludeTables.split(",")` keeps surrounding whitespace, so
`"a, b"` silently fails to exclude `b`. A `.trim()` per element would prevent a
class of hard-to-spot config errors.
2. **Error ordering.** `generateCreateTableCmds` collects `noPrimaryKeyTables` while
iterating, but throws only *after* the loop
(`StreamingJobUtils.java` L469-472). Tables processed before the failure have already
been created in Doris, so a failed `CREATE JOB` leaves partial state that must be
cleaned up manually. Failing fast, or rolling back created tables, would be friendlier.

### Use case

For whole-database sync this becomes hard to maintain. In our production database
(285 base tables) there are 16+ manually-created backup/archive tables following
recognizable naming patterns:

```
sys_config_bak_20260730
sys_config_bak_20260803_codex
sys_config_bak_20260804
sys_config_bak_codex_progress_20260810
sys_config_bak_mcp_20260812
sys_config_bak_progress_20260813
sys_config_bak_progress_202608132
sys_config_bak_xie_20260806
sys_config_backup_20260710_codex_mcp_skill_sync_fix
sys_config_backup_20260710_codex_mcp_workflow
plugin_config_backup_20260721
sys_credit_record_archive_bak_20260805
sys_dept_permission_scope_bak_20260509
sys_permission_bak_20260808
sys_permission_model_bak_20260820
sys_role_bak_20260808
```

With exact matching, every one of these must be enumerated, and — more importantly —
**new backup tables created upstream after the job was created are silently picked up**.
Since `exclude_tables` cannot be changed by `ALTER JOB`, keeping the exclusion list current
requires dropping and recreating the job (and carefully preserving the binlog offset).

A pattern like `sys_config_bak_.*` or `*_bak_*` would express the intent once and stay correct.

### Related issues

_No response_

### Are you willing to submit PR?

- [x] Yes I am willing to submit a PR!

### Code of Conduct

- [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)

Contributor guide

Open the contributing guide

Research direction

Start in fe/fe-core/src/main/java/org/apache/doris/job/util/StreamingJobUtils.java, especially the exclude/include matching around L378-402 and generateCreateTableCmds around L469-472. Trace how CREATE JOB ... ON STREAMING FROM MYSQL handles table patterns and failures, then define completion as documented full-match behavior with backward-compatible exact names, trimmed entries, and no unintended partial state after an error.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, mysql
Domain
databases
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
58/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.