[Feature] MySQL CDC: support regex (or wildcard) in exclude_tables / include_tables
- Dominant language
- Java
- Stars
- 15.9k
- Forks
- 3.9k
- Avg merge
- 2d 23h
- Merged PRs (30d)
- 520
Description
### Search before asking
- [x] I had searched in the [issues](https://github.com/apache/doris/issues?q=is%3Aissue) and found no similar issues.
### Description
I would like to request adding regular expression (or wildcard) support for the `exclude_tables` and `include_tables` configurations in `CREATE JOB ... ON STREAMING FROM MYSQL`.
Currently, `exclude_tables` / `include_tables` perform exact string matching only:
`fe/fe-core/src/main/java/org/apache/doris/job/util/StreamingJobUtils.java`
```java
excludeTablesList = Arrays.asList(excludeTables.split(",")); // L378-380
// ...
if (includeTablesList.isEmpty() && !excludeTablesList.isEmpty() && excludeTablesList.contains(table)) { // L401-402
continue;
}
```
### Solution
Support regex (or at minimum glob-style wildcards) in both `exclude_tables` and
`include_tables`. Flink CDC's `tables` option already accepts regex
(e.g. `app_db.\.*`), so this would also align the two code paths.
Suggested semantics:
- Keep backward compatibility: a pattern with no regex metacharacters behaves exactly as today.
- Match against the bare table name (as today), not the qualified name, to avoid ambiguity.
- Document whether the match is full-match or partial-match (`matches()` vs `find()`) —
full-match is less surprising.
Two related nits found while investigating, worth fixing in the same change:
1. **No trimming.** `excludeTables.split(",")` keeps surrounding whitespace, so
`"a, b"` silently fails to exclude `b`. A `.trim()` per element would prevent a
class of hard-to-spot config errors.
2. **Error ordering.** `generateCreateTableCmds` collects `noPrimaryKeyTables` while
iterating, but throws only *after* the loop
(`StreamingJobUtils.java` L469-472). Tables processed before the failure have already
been created in Doris, so a failed `CREATE JOB` leaves partial state that must be
cleaned up manually. Failing fast, or rolling back created tables, would be friendlier.
### Use case
For whole-database sync this becomes hard to maintain. In our production database
(285 base tables) there are 16+ manually-created backup/archive tables following
recognizable naming patterns:
```
sys_config_bak_20260730
sys_config_bak_20260803_codex
sys_config_bak_20260804
sys_config_bak_codex_progress_20260810
sys_config_bak_mcp_20260812
sys_config_bak_progress_20260813
sys_config_bak_progress_202608132
sys_config_bak_xie_20260806
sys_config_backup_20260710_codex_mcp_skill_sync_fix
sys_config_backup_20260710_codex_mcp_workflow
plugin_config_backup_20260721
sys_credit_record_archive_bak_20260805
sys_dept_permission_scope_bak_20260509
sys_permission_bak_20260808
sys_permission_model_bak_20260820
sys_role_bak_20260808
```
With exact matching, every one of these must be enumerated, and — more importantly —
**new backup tables created upstream after the job was created are silently picked up**.
Since `exclude_tables` cannot be changed by `ALTER JOB`, keeping the exclusion list current
requires dropping and recreating the job (and carefully preserving the binlog offset).
A pattern like `sys_config_bak_.*` or `*_bak_*` would express the intent once and stay correct.
### Related issues
_No response_
### Are you willing to submit PR?
- [x] Yes I am willing to submit a PR!
### Code of Conduct
- [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
Contributor guide
Research direction
Start in fe/fe-core/src/main/java/org/apache/doris/job/util/StreamingJobUtils.java, especially the exclude/include matching around L378-402 and generateCreateTableCmds around L469-472. Trace how CREATE JOB ... ON STREAMING FROM MYSQL handles table patterns and failures, then define completion as documented full-match behavior with backward-compatible exact names, trimmed entries, and no unintended partial state after an error.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, mysql
- Domain
- databases
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 58/100