aws / aws/aws-advanced-python-wrapper
failover/failover_v2 (sync) lose the driver exception type for errors
- Dominant language
- Python
- Stars
- 98
- Forks
- 22
- Avg merge
- 1d 9h
- Merged PRs (30d)
- 5
Description
### Describe the bug
The sync `failover` and `failover_v2` plugins wrap every exception raised from `execute()` in `AwsWrapperError`, including exceptions that have nothing to do with failover.
https://github.com/aws/aws-advanced-python-wrapper/blob/d23b2ebcee6847326d1718e150ff309b0d388bf3/aws_advanced_python_wrapper/failover_v2_plugin.py#L198-L208
A unique constraint violation (MySQL 1062) is not a failover condition, but it is wrapped all the same, and the original driver exception's class identity is lost. SQLAlchemy, Django and Celery all classify database errors by exception class, so their standard error handling paths stop matching.
This also makes the sync plugins inconsistent with other AWS Advanced Wrapper implementation, and with this repository's own async failover plugin, all of which re-raise the original exception when no failover was triggered.
### Expected Behavior
A driver exception that does not trigger failover (such as MySQL 1062) should be classified as `sqlalchemy.exc.IntegrityError` and the standard idiom below works:
```python
try:
session.add(obj)
session.commit()
except IntegrityError: # the loser of a unique-key race should land here
session.rollback()
return session.query(Model).filter_by(**kwargs).one()
```
### What plugins are used? What other connection properties were set?
`plugins=aurora_connection_tracker,failover_v2`, `wrapper_dialect=aurora-mysql`, `cluster_id=`, URL `mysql+aws_wrapper_mysqlconnector://`, SQLAlchemy 2.0. Also reproduces with `plugins=failover`.
### Current Behavior
MySQL 1062 surfaces as a generic `sqlalchemy.exc.DBAPIError` instead of `sqlalchemy.exc.IntegrityError`:
```
sqlalchemy.exc.DBAPIError: (aws_advanced_python_wrapper.AwsWrapperError)
[Failover] Detected an exception while executing a command:
1062 (23000): Duplicate entry 'example-name' for key 'table.column
[SQL: INSERT INTO table (column, ...) VALUES (%s, %s, ...)]
```
SQLAlchemy picks the `sqlalchemy.exc` class by walking the raised exception's base classes and matching their names. None of `AwsWrapperError`'s bases match a name that `sqlalchemy.exc` exports, so classification falls back to the generic `DBAPIError`.
Three separate surfaces are affected:
1. SQLAlchemy. Any `except IntegrityError:` recovery path stops matching. Previously working race handling turns into a 500.
2. Django. `wrap_database_errors` matches against `mysql.connector`'s error classes, which `AwsWrapperError` does not subclass, so nothing matches and it propagates raw. Django's own `QuerySet.get_or_create` no longer works.
3. Celery database result backend. Its retry list (`DatabaseError`, `InterfaceError` and friends) sits below `DBAPIError`, so result writes never retry.
### Reproduction Steps
No Aurora cluster is required, since the problem is exception classification rather than failover itself.
```python
import aws_advanced_python_wrapper
from aws_advanced_python_wrapper.errors import AwsWrapperError
from mysql.connector import errors as mysql_errors
from sqlalchemy import exc as sa_exc
driver_error = mysql_errors.IntegrityError(
"1062 (23000): Duplicate entry 'example-name' for key 'table.column'"
)
# same shape as what _deal_with_original_exception raises
wrapped = AwsWrapperError(
"[Failover] Detected an exception while executing a command", driver_error
)
result = sa_exc.DBAPIError.instance(
"INSERT INTO table (...) VALUES (...)", {},
wrapped, aws_advanced_python_wrapper.Error,
)
print(type(result))
# actual:
# expected:
# control: the unwrapped driver error is classified correctly
print(type(sa_exc.DBAPIError.instance("INSERT ...", {}, driver_error,
mysql_errors.Error)))
#
```
End to end, this happens whenever two processes insert the same unique key concurrently against Aurora MySQL with the plugin chain above. Before the wrapper was introduced, `except IntegrityError:` absorbed the loser and re-read the winning row. After it, the exception arrives as `DBAPIError` and the handler is bypassed.
### Possible Solution
Two directions, and I am happy to contribute either.
### Option 1: keep raising `AwsWrapperError`, but carry the driver error's PEP-249 category
Declare one class per PEP-249 category, inheriting from both `AwsWrapperError` and the wrapper's own `pep249` class, and pick the matching one when wrapping a driver error.
```python
# errors.py
class AwsWrapperIntegrityError(AwsWrapperError, pep249.IntegrityError): ...
class AwsWrapperOperationalError(AwsWrapperError, pep249.OperationalError): ...
# and DataError, InternalError, NotSupportedError, ProgrammingError,
# InterfaceError, DatabaseError
```
`except AwsWrapperError:` keeps working, and SQLAlchemy now resolves `IntegrityError`, itself a subclass of `DBAPIError`, so `except DBAPIError:` keeps working too. Category selection can reuse the ordering already in `_normalize_driver_error`. (Failover signals carry no `driver_error` and never take this path.)
### Option 2: re-raise the original exception, matching the other implementations
```python
def _deal_with_original_exception(self, original_exception: Exception) -> None:
if (self._last_exception_dealt_with != original_exception and
self._should_exception_trigger_connection_switch(original_exception)):
...
raise AwsWrapperError(...) from original_exception
raise
```
Consistent with the async plugin and the other wrappers, and removes code rather than adding it. Breaking for 3.x users who catch `AwsWrapperError` around ordinary query errors.
### Django
Neither option fixes Django, which matches against `mysql.connector`'s classes. The smallest fix is to override `wrap_database_errors` in the wrapper's Django backend: let `FailoverError` through untouched, and when the exception carries a `driver_error`, classify using that exception's class. Raw driver errors on plugin chains that do not re-wrap are unaffected.
### Additional Information/Context
Other implementations re-raise the original exception when the error does not trigger failover, replacing it only when failover actually happens:
- JDBC wrapper, both `failover` and `failover2`
- .NET data provider wrapper
- Node.js wrapper, both `failover` and `failover2`
- Go wrapper
- This repository's own async failover plugin
The sync Python plugins are the only place that wraps in the non-failover case.
`docs/using-the-python-wrapper/SqlAlchemySupport.md` states that target driver exceptions are not remapped and flow through SQLAlchemy's dialect specific classification unchanged.
With a failover plugin in the chain that is not what happens, since driver exceptions are always re-wrapped. That section will need updating alongside whichever fix is done.
### The AWS Advanced Python Wrapper version used
3.0.0 and 3.1.0
### python version used
Python 3.11.11
### Operating System and version
Amazon Linux 2023 (container), also reproduced on macOS 15
Contributor guide
Assessment
This issue has not been assessed yet.