aws / aws/aws-secretsmanager-jdbc

using MySQL `autoReconnect=true` configuration results in unhandled access denied errors

Open
#144 2 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Java
Stars
196
Forks
87
PR merge metrics
No merged PRs in 30d

Description

Took some time tracking down the root cause, but I was getting `com.mysql.cj.exceptions.CJException: Access denied for user` while using the secrets manager MySQL driver for MySQL driver 8.0.31. This is a problem because [the error handling for the AWSSecretsManagerMySQLDriver](https://github.com/aws/aws-secretsmanager-jdbc/blob/765270e62889a42a207dc3e4591af60058756040/src/main/java/com/amazonaws/secretsmanager/sql/AWSSecretsManagerMySQLDriver.java#L108-L114) expects a SQLException but the CJException is not a SQLException subclass, so these credential issues were not being automatically handled by the driver.

I found that the cause was using the `autoReconnect=true` JDBC parameter. When using this parameter, the MySQL driver ends up taking a separate code from `autoReconnect=false`:

```
java.sql.SQLNonTransientConnectionException: Could not create connection to database server. Attempted reconnect 3 times. Giving up.
at com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:110)
at com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:97)
at com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:89)
at com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:63)
at com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:73)
at com.mysql.cj.jdbc.ConnectionImpl.connectWithRetries(ConnectionImpl.java:898)
at com.mysql.cj.jdbc.ConnectionImpl.createNewIO(ConnectionImpl.java:823)
at com.mysql.cj.jdbc.ConnectionImpl.(ConnectionImpl.java:448)
at com.mysql.cj.jdbc.ConnectionImpl.getInstance(ConnectionImpl.java:241)
at com.mysql.cj.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:198)
...
Caused by: com.mysql.cj.exceptions.CJException: Access denied for user 'test_user1671740990998'@'172.17.0.1' (using password: YES)
at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:490)
at com.mysql.cj.exceptions.ExceptionFactory.createException(ExceptionFactory.java:61)
at com.mysql.cj.exceptions.ExceptionFactory.createException(ExceptionFactory.java:105)
at com.mysql.cj.exceptions.ExceptionFactory.createException(ExceptionFactory.java:151)
at com.mysql.cj.exceptions.ExceptionFactory.createException(ExceptionFactory.java:129)
at com.mysql.cj.protocol.a.NativeProtocol.checkErrorMessage(NativeProtocol.java:848)
at com.mysql.cj.protocol.a.NativeProtocol.checkErrorMessage(NativeProtocol.java:770)
at com.mysql.cj.protocol.a.NativeProtocol.checkErrorMessage(NativeProtocol.java:738)
at com.mysql.cj.protocol.a.NativeProtocol.checkErrorMessage(NativeProtocol.java:155)
at com.mysql.cj.protocol.a.NativeAuthenticationProvider.proceedHandshakeWithPluggableAuthentication(NativeAuthenticationProvider.java:472)
at com.mysql.cj.protocol.a.NativeAuthenticationProvider.connect(NativeAuthenticationProvider.java:212)
at com.mysql.cj.protocol.a.NativeProtocol.connect(NativeProtocol.java:1433)
at com.mysql.cj.NativeSession.connect(NativeSession.java:133)
at com.mysql.cj.jdbc.ConnectionImpl.connectWithRetries(ConnectionImpl.java:842)
... 82 more

```

When `autoReconnect=false`, the `com.mysql.cj.jdbc.ConnectionImpl.createNewIO` method calls `com.mysql.cj.jdbc.ConnectionImpl.connectOneTryOnly` instead of `com.mysql.cj.jdbc.ConnectionImpl.connectWithRetries`. The `com.mysql.cj.jdbc.ConnectionImpl.connectWithRetries` method ends up throwing a SQLNonTransientConnectionException containing a CJException with a vendorCode equal to the expected error code `1045`.

So, the immediate fix I've found is to just not use `autoReconnect=true`, turns out [its usage is not recommended by MySQL anyway](https://dev.mysql.com/doc/connector-j/8.0/en/connector-j-connp-props-high-availability-and-clustering.html#:~:text=The%20use%20of%20this%20feature,from%20dead%20and%20stale%20connections). But it seems like [others have found the error detection to be problematic](https://github.com/aws/aws-secretsmanager-jdbc/pull/136) and the error handling here could be improved to unwrap exceptions and check for CJExceptions as well as SQLExceptions

Contributor guide

Open the contributing guide

Research direction

Start in src/main/java/com/amazonaws/secretsmanager/sql/AWSSecretsManagerMySQLDriver.java at the error-handling code around lines 108-114, then compare it with the MySQL Connector/J paths described for autoReconnect=true and autoReconnect=false. Use the behavior in this issue and pull request #136 as references; done means credential errors from the reconnect path are detected and handled by the driver.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, java, mysql
Domain
backend, databases, security
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.