apache / apache/druid

After Mysql (aws aurora) failover, druid is still connected to the same instace (now readonly instance)

Open
#15,638 2 comments 1 reaction 0 assignees View on GitHub
Area - Metadata Bug
Dominant language
Java
Stars
14.1k
Forks
3.8k
Avg merge
2d 58m
Merged PRs (30d)
233

Description

After Mysql (aws aurora) failover, druid is still connected to the same instace (now readonly instance)

### Affected Version
27.0.0

### Description

- Cluster size: 15 node
- Configurations in use: using mysql metastore with aws aurora, the URL is the writer url of the aurora database cluster which will be switched to the current writer instance (the ip address will be updated when failover happens).
- Steps to reproduce the problem: druid cluster with mysql metadatastore deployed. aurora mysql with at least 2 instances. Use the writer endpoint in druid configurations. Then fail over the cluster to the second node.
- The error message or stack traces encountered. Providing more context, such as nearby log messages or even entire logs, can be helpful.
![image](https://github.com/apache/druid/assets/17374492/2f14a272-d1a3-4cce-a8be-ea6e587b89d4)

I believe there should be a catch for this exception that renews the mysql connection pool. This will allow druid to always connect to the writer instance on highly available environments.

Extra info after communicating with AWS support:
In the event of mysql failover the writer endpoint ip address will change to the new writer instance ip address. This instance was previously reader. The old writer instance will become a reader instance. In druid's case after the mysql failover, druid is still using the cached ip address of the previous writer instance(which is now read replica aka read only). In order to fix this druid needs to resolve the endpoint to ip address again and use the new ip address.

Another option to fix this the easy way is to add support to aurora mysql via [AWS official mysql driver for java](https://aws.amazon.com/about-aws/whats-new/2022/03/amazon-web-services-jdbc-driver-mysql-failover/).

> The [Amazon Web Services (AWS) Java (JDBC) Driver for MySQL](https://awslabs.github.io/aws-mysql-jdbc/) is now generally available for use with your Amazon RDS or Amazon Aurora MySQL-compatible edition database clusters. This database driver minimizes failover time by monitoring database cluster status and caching the cluster’s topology. The cache is then used to improve DNS name resolution speed if a node fails, reducing failover time from minutes to seconds.

> The MySQL JDBC driver is based on the [MySQL Connector/J](https://github.com/mysql/mysql-connector-j) and supports MySQL deployments. It is drop-in compatible with the community MySQL driver and is installed with Maven, Gradle, or by .jar file. After installing the required file, your client requires simple changes to the connection string to use the new driver.

Contributor guide

Open the contributing guide

Research direction

Reproduce the reported setup with Druid 27.0.0, a MySQL metadata store, an AWS Aurora cluster with at least two instances, and the writer endpoint, then trace how the connection pool handles failover and the resulting read-only connection. Compare renewing the pool after the failure with using the AWS MySQL JDBC driver; done means Druid reconnects to the new writer after failover.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, java, mysql
Domain
backend, databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.