apache / apache/rocketmq-externals

[rocketmq-connect-runtime] possible enhancements recorder

Open
#540 9 comments 3 reactions 0 assignees View on GitHub
enhancement soc
Dominant language
Java
Stars
4.6k
Forks
3k
Avg merge
2h 33m
Merged PRs (30d)
1

Description

The issue tracker is **ONLY** used for bug report and feature request.

Any question or RocketMQ proposal please use our [mailing lists](http://rocketmq.apache.org/about/contact/).
**FEATURE REQUEST**
This is a recorder for issues I've met when deploying rocketmq-connect-runtime, and there are not many core-related or performance enhancements I can observe now, so I'll just put some simple enhancements as candidates.

- [x] runtime stopAll doesn't stop immediately, log system still outputs stale messages. And after deleting the configuration files the runtime connect still retrieves the old configuration. Old configuration can not be removed and will always appear in the log.
- [x] The log systems are printing some very detailed logs, we can enhance the log system of runtime by providing multiple log levels so developers/users can pick the right amount of logs needed to debug/develop/troubleshooting.
- [x] fix up typos along developing
- [x] jdbc sink connector is observing route topic changes, but I intuitively feel this dynamic routing management can be moved to runtime, need to discuss with mentors.
- [x] currently if user wants to change a connector configuration, they need to use either web browser or `curl` command. However if user use `curl` command they need to encode the config json to a url-encoded format. We could write some simple script or CLI to help user deploy connectors more easily, for example it will probably look like `rcadmin createConnector -n -f jdbc-sink-connector-config.json`
- [x] taskConfigs() could return null value, and this exception should be handled if no task configs are generated by connector.
- [x] JdbcSinkTask#stop() will not close the connection if the connection was never established. When sinkTask tries to do `connection = dataSource.getConnection();` and if the connection is invalid, this function will retry multiple time and throws error
- [x] runtime have a race condition when a task should be stopped but another `pullMessageFromQueues` can still be executed, result in `org.apache.rocketmq.client.exception.MQClientException: The consumer is not in running status, SHUTDOWN_ALREADY` exception, see detailed down below.

Contributor guide

No contributing guide indexed for this repository

Research direction

The issue lists several runtime and connector entry points, including stopAll, taskConfigs(), JdbcSinkTask#stop(), and pullMessageFromQueues, plus a possible rcadmin CLI. Start by separating the checked enhancement candidates and locating these entry points in the rocketmq-connect-runtime code. Done criteria are not defined because the issue combines multiple proposals and contains no single remaining task.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
backend
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.