apache / apache/airflow

Block migrations on MySQL when using bad collations/encoding

Open
#19,289 6 comments 2 reactions 0 assignees View on GitHub
_eol_affected_version:2.2 area:core area:upgrade kind:bug Stale Bug Report
Dominant language
Python
Stars
46.9k
Forks
17.8k
Avg merge
2d 10h
Merged PRs (30d)
483

Description

### Apache Airflow version

main (development)

### Operating System

MacOS 11.6

### Versions of Apache Airflow Providers

_No response_

### Deployment

Virtualenv installation

### Deployment details

_No response_

### What happened

Currently we happily attempt to migrate the database regardless of what collation and encoding is used.
This can cause migrations to fail unexpectedly, and with MySQL's lack of DDL transactions can leave the database
in a half-migrated broken state (see #18939 and https://github.com/apache/airflow/issues/18939#issuecomment-953395944).

### What you expected to happen

We should identify when we have bad values for encoding/collation, block migrations from happening, and offer detailed instructions on how to fix it.

### How to reproduce

Create a MySQL database without specifying the encoding/collation as documented in [Setting up a MySQL database](https://airflow.apache.org/docs/apache-airflow/stable/howto/set-up-database.html#setting-up-a-mysql-database).

### Anything else

_No response_

### Are you willing to submit PR?

- [X] Yes I am willing to submit a PR!

### Code of Conduct

- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the problem using the MySQL setup instructions in the “Setting up a MySQL database” documentation, with an unspecified encoding or collation. Trace the migration entry point and determine where invalid values can be detected before migrations run. Done means invalid MySQL settings block migration and provide detailed correction instructions without leaving a half-migrated database.

Written by the indexing model from the issue text.

Assessment

Tech stack
mysql, python
Domain
databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.