apache / apache/doris-flink-connector

[Bug] Doris include table pattern null

Open
#294 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
385
Forks
279
Avg merge
2d 2h
Merged PRs (30d)
6

Description

### Search before asking

- [X] I had searched in the [issues](https://github.com/apache/incubator-doris/issues?q=is%3Aissue) and found no similar issues.

### Version

Doris flink connector: flink-doris-connector-1.17-1.5.0

### What's Wrong?

When I use `--single-sink` flag to start my flink job, the task will report error:

```text
Caused by: java.lang.IllegalArgumentException: Can't find any matched tables, please check your configured database-name: [svc_dw] and table-name: [((svc_dw)\.(null))(?!(svc_dw\.(upload_histories|supported_schemas))$)]
```

I try to figure out the question and I found the [code](https://github.com/apache/doris-flink-connector/blob/d0c1d7c61db56c891071de6cb4d5bcea26835e15/flink-doris-connector/src/main/java/org/apache/doris/flink/tools/cdc/DatabaseSync.java#L308-L325) that causes the question:
```java
protected String getSyncTableList(List syncTables) {
if (!singleSink) {
return syncTables.stream()
.map(v -> getTableListPrefix() + "\\." + v)
.collect(Collectors.joining("|"));
}

// includingTablePattern and ^excludingPattern
String includingPattern =
String.format("(%s)\\.(%s)", getTableListPrefix(), includingTables);
if (StringUtils.isNullOrWhitespaceOnly(excludingTables)) {
return includingPattern;
} else {
String excludingPattern =
String.format("?!(%s\\.(%s))$", getTableListPrefix(), excludingTables);
return String.format("(%s)(%s)", includingPattern, excludingPattern);
}
}
```

If I use `--excluding-tables` flag but not use `--including-tables` flag with `--single-sink` flag,the mysql flink cdc source will get the `--table-name` with value: `((svc_dw)\.(null))(?!(svc_dw\.(upload_histories|supported_schemas))$)`

### What You Expected?

Start the flink job successfully with `--single-sink` and `--excluding-tables` flags.

### How to Reproduce?

Start a flink job with `--single-sink` and `--excluding-tables` flags and without `--including-tables` flag.

### Anything Else?

_No response_

### Are you willing to submit PR?

- [X] Yes I am willing to submit a PR!

### Code of Conduct

- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)

Contributor guide

Open the contributing guide

Research direction

Start in flink-doris-connector/src/main/java/org/apache/doris/flink/tools/cdc/DatabaseSync.java at getSyncTableList, referenced in the issue. Reproduce with --single-sink and --excluding-tables without --including-tables, then verify the generated table pattern no longer contains null and the Flink job starts successfully.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, mysql
Domain
backend, data-engineering
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Stale
Clarity
Clearly specified
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.