apache / apache/rocketmq-externals

[rocketmq-connect-runtime] Connector not recomputing task configs after connector config changes

Open
#608 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
4.6k
Forks
3k
Avg merge
2h 33m
Merged PRs (30d)
1

Description

The issue tracker is **ONLY** used for bug report and feature request.

Any question or RocketMQ proposal please use our [mailing lists](http://rocketmq.apache.org/about/contact/).

**BUG REPORT**

1. Please describe the issue you observed:

```
Set stoppedConnector = new HashSet<>();
for (WorkerConnector workerConnector : workingConnectors) {
String connectorName = workerConnector.getConnectorName();
ConnectKeyValue keyValue = connectorConfigs.get(connectorName);
if (null == keyValue || 0 != keyValue.getInt(RuntimeConfigDefine.CONFIG_DELETED)) {
workerConnector.stop();
log.info("Connector {} stop", workerConnector.getConnectorName());
stoppedConnector.add(workerConnector);
} else if (!keyValue.equals(workerConnector.getKeyValue())) {
// I think we should trigger a task recompute, like if task parallelism changed, we should spawn more tasks
workerConnector.reconfigure(keyValue);
}
}
workingConnectors.removeAll(stoppedConnector);

```

- What did you do (The steps to reproduce)?

- What did you expect to see?

- What did you see instead?

2. Please tell us about your environment:

3. Other information (e.g. detailed explanation, logs, related issues, suggestions how to fix, etc):

**FEATURE REQUEST**

1. Please describe the feature you are requesting.

2. Provide any additional detail on your proposed use case for this feature.

2. Indicate the importance of this issue to you (blocker, must-have, should-have, nice-to-have). Are you currently using any workarounds to address this issue?

4. If there are some sub-tasks using -[] for each subtask and create a corresponding issue to map to the sub task:

- [sub-task1-issue-number](example_sub_issue1_link_here): sub-task1 description here,
- [sub-task2-issue-number](example_sub_issue2_link_here): sub-task2 description here,
- ...

Contributor guide

No contributing guide indexed for this repository

Research direction

Locate the connector-management loop in rocketmq-connect-runtime and inspect WorkerConnector.reconfigure and the handling of workingConnectors. Reproduce a connector configuration change, especially a task-parallelism change, and verify that the resulting task configurations are recomputed and the running tasks reflect the update.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.