aws / aws/aws-secretsmanager-jdbc
using MySQL `autoReconnect=true` configuration results in unhandled access denied errors
- Dominant language
- Java
- Stars
- 196
- Forks
- 87
- PR merge metrics
- No merged PRs in 30d
Description
Took some time tracking down the root cause, but I was getting `com.mysql.cj.exceptions.CJException: Access denied for user` while using the secrets manager MySQL driver for MySQL driver 8.0.31. This is a problem because [the error handling for the AWSSecretsManagerMySQLDriver](https://github.com/aws/aws-secretsmanager-jdbc/blob/765270e62889a42a207dc3e4591af60058756040/src/main/java/com/amazonaws/secretsmanager/sql/AWSSecretsManagerMySQLDriver.java#L108-L114) expects a SQLException but the CJException is not a SQLException subclass, so these credential issues were not being automatically handled by the driver.
I found that the cause was using the `autoReconnect=true` JDBC parameter. When using this parameter, the MySQL driver ends up taking a separate code from `autoReconnect=false`:
```
java.sql.SQLNonTransientConnectionException: Could not create connection to database server. Attempted reconnect 3 times. Giving up.
at com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:110)
at com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:97)
at com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:89)
at com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:63)
at com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:73)
at com.mysql.cj.jdbc.ConnectionImpl.connectWithRetries(ConnectionImpl.java:898)
at com.mysql.cj.jdbc.ConnectionImpl.createNewIO(ConnectionImpl.java:823)
at com.mysql.cj.jdbc.ConnectionImpl.(ConnectionImpl.java:448)
at com.mysql.cj.jdbc.ConnectionImpl.getInstance(ConnectionImpl.java:241)
at com.mysql.cj.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:198)
...
Caused by: com.mysql.cj.exceptions.CJException: Access denied for user 'test_user1671740990998'@'172.17.0.1' (using password: YES)
at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:490)
at com.mysql.cj.exceptions.ExceptionFactory.createException(ExceptionFactory.java:61)
at com.mysql.cj.exceptions.ExceptionFactory.createException(ExceptionFactory.java:105)
at com.mysql.cj.exceptions.ExceptionFactory.createException(ExceptionFactory.java:151)
at com.mysql.cj.exceptions.ExceptionFactory.createException(ExceptionFactory.java:129)
at com.mysql.cj.protocol.a.NativeProtocol.checkErrorMessage(NativeProtocol.java:848)
at com.mysql.cj.protocol.a.NativeProtocol.checkErrorMessage(NativeProtocol.java:770)
at com.mysql.cj.protocol.a.NativeProtocol.checkErrorMessage(NativeProtocol.java:738)
at com.mysql.cj.protocol.a.NativeProtocol.checkErrorMessage(NativeProtocol.java:155)
at com.mysql.cj.protocol.a.NativeAuthenticationProvider.proceedHandshakeWithPluggableAuthentication(NativeAuthenticationProvider.java:472)
at com.mysql.cj.protocol.a.NativeAuthenticationProvider.connect(NativeAuthenticationProvider.java:212)
at com.mysql.cj.protocol.a.NativeProtocol.connect(NativeProtocol.java:1433)
at com.mysql.cj.NativeSession.connect(NativeSession.java:133)
at com.mysql.cj.jdbc.ConnectionImpl.connectWithRetries(ConnectionImpl.java:842)
... 82 more
```
When `autoReconnect=false`, the `com.mysql.cj.jdbc.ConnectionImpl.createNewIO` method calls `com.mysql.cj.jdbc.ConnectionImpl.connectOneTryOnly` instead of `com.mysql.cj.jdbc.ConnectionImpl.connectWithRetries`. The `com.mysql.cj.jdbc.ConnectionImpl.connectWithRetries` method ends up throwing a SQLNonTransientConnectionException containing a CJException with a vendorCode equal to the expected error code `1045`.
So, the immediate fix I've found is to just not use `autoReconnect=true`, turns out [its usage is not recommended by MySQL anyway](https://dev.mysql.com/doc/connector-j/8.0/en/connector-j-connp-props-high-availability-and-clustering.html#:~:text=The%20use%20of%20this%20feature,from%20dead%20and%20stale%20connections). But it seems like [others have found the error detection to be problematic](https://github.com/aws/aws-secretsmanager-jdbc/pull/136) and the error handling here could be improved to unwrap exceptions and check for CJExceptions as well as SQLExceptions
Contributor guide
Research direction
Start in src/main/java/com/amazonaws/secretsmanager/sql/AWSSecretsManagerMySQLDriver.java at the error-handling code around lines 108-114, then compare it with the MySQL Connector/J paths described for autoReconnect=true and autoReconnect=false. Use the behavior in this issue and pull request #136 as references; done means credential errors from the reconnect path are detected and handled by the driver.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, java, mysql
- Domain
- backend, databases, security
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100