apache / apache/dolphinscheduler

[DSIP-97][API/Worker] datasource supports multiple driver versions

Open
#17,764 3 comments 0 reactions 0 assignees View on GitHub
discussion DSIP
Dominant language
Java
Stars
14.5k
Forks
5.1k
Avg merge
1d 21h
Merged PRs (30d)
29

Description

### Search before asking

- [x] I had searched in the [DSIP](https://github.com/apache/dolphinscheduler/issues/14102) and found no similar DSIP.

### Motivation

Reason: The current data source plugin does not have a class loading isolation mechanism, which makes it impossible to support multiple versions of the data source driver simultaneously
Implementation goal: When users create data sources, they can choose the data source driven version for flexible customization

### Design Detail
1. Front end:
(1) Add data source driver version information when creating, modifying data sources
(2) Call the backend interface to query the data source version list display

Image

2. Backend:
(1) Add a new database driver version configuration table, provide an interface to query the data source version list, match and query the driver version configuration table based on the version number to query the corresponding version of the driver package (main version enumeration)

CREATE TABLE `t_ds_datasource_driver_version_config` (
`id` int(11) NOT NULL AUTO_INCREMENT COMMENT 'key',
`type` tinyint(4) NOT NULL COMMENT 'data source type: 0:mysql,1:postgresql,2:hive,3:spark',
`version` varchar(255) COLLATE utf8_bin DEFAULT NULL COMMENT 'version'
`driver_class_name` varchar(255) COLLATE utf8_bin DEFAULT NULL COMMENT 'driver_class_name',
`driver_jar_name` varchar(255) COLLATE utf8_bin DEFAULT NULL COMMENT 'driver_jar_name'
PRIMARY KEY (`id`),
UNIQUE KEY `t_ds_datasource_driver_type_version` (`type`,`version`)
) ENGINE=InnoDB AUTO_INCREMENT=1 DEFAULT CHARSET=utf8 COLLATE=utf8_bin;

(2) Add version field to data source connection parameters

{
"user": "datawings",
"password": "xxx"
"address": "jdbc:mysql://127.0.0.1:50001",
"database": "test",
"jdbcUrl": "jdbc:mysql://127.0.0.1:50001/test",
"driverClassName": "com.mysql.cj.jdbc.Driver",
"version":"8.0.33",
"validationQuery": "select 1"
}

(3) When performing tasks with different data sources and creating database connections, use a custom class loader to dynamically load the corresponding version of the driver package

Image

a. POM configuration: Package multiple versions of driver packages into a specified directory (such as lib/drivers/) to avoid Maven dependency mediation coverage;
b. Dependency isolation: Manually copy multiple versions of drivers to different subdirectories through Maven dependency plugin, instead of relying on Maven for automatic dependency management;
c. Run time loading: By using a custom class loader to load a specified version of the driver package, multiple versions can be isolated for use. Each version of the driver uses an independent URLClassLoader, and the parent loader is set to null to avoid JVM default class loaders loading conflicting classes;

### Compatibility, Deprecation, and Migration Plan

Use the default driver package to execute existing tasks

### Test Plan

(1) Test connection success when creating different versions of data sources
(2) The same workflow uses different driver versions of data sources, and the workflow executes normally

### Code of Conduct

- [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)

Contributor guide

Open the contributing guide

Research direction

Start by reading the API/Worker data-source creation and task connection flows, then trace how driver dependencies are packaged and loaded. The design identifies a driver-version configuration table, a version field in connection parameters, and isolated class loading; done means the listed connection and workflow tests pass for multiple driver versions.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, mysql, postgresql, spark
Domain
api, backend, build-system, databases
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.