sqlalchemy / sqlalchemy/alembic

Add support for mysql ALTER TABLE with multiple alter_specification

Open
#271 5 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

batch migrations feature
Dominant language
Python
Stars
4.4k
Forks
375
PR merge metrics
No merged PRs in 30d

Description

Migrated issue, originally created by vitaly numenta

I'd like to start by professing my appreciation for alembic (and sqlalchemy) - great job!

I don't know if I picked the correct Component for this, so please update if needed.

Recently, I performed a multi-column alembic migration on a ten-million-row mysql table that took a whopping 17 hours to complete. As I was watching the table, I'd see each new column showing up piecemeal every 2-4 hours.

It turns out that mysql implements most ALTER TABLE statements by first implicitly making a full copy of the entire table, which explains why it took so long to add several new columns to the table using individual ALTER TABLE command for each ADD COLUMN as generated by alembic. The original table is write-locked during the entire operation, so it was effectively off-line for 17 hours.

mysql supports bundling multiple column changes inside a single ALTER TABLE statement, which means only one full implicit copy of the entire table, instead of multiple copes:

ALTER TABLE tbl_name alter_specification, alter_specification, ...

for example: ALTER TABLE addresses DROP COLUMN c, DROP COLUMN d, ADD COLUMN mobile VARCHAR(40);

For performance, it's important to be able to do the above in in alembic as well. I was hoping that the new batch_alter_table would do that, but it has a different model in mind (designed to work around sqlite's limitations).

However, instead of the single ALTER TABLE statement, alembic generates multiple ALTER TABLE statements instead like this:
ALTER TABLE addresses DROP COLUMN c;
ALTER TABLE addresses DROP COLUMN d;
ALTER TABLE addresses ADD COLUMN mobile VARCHAR(40);

Per http://dev.mysql.com/doc/refman/5.1/en/alter-table.html: "In most cases, ALTER TABLE makes a temporary copy of the original table. MySQL waits for other operations that are modifying the table, then proceeds. It incorporates the alteration into the copy, deletes the original table, and renames the new one. While ALTER TABLE is executing, the original table is readable by other sessions. Updates and writes to the table that begin after the ALTER TABLE operation begins are stalled until the new table is ready..."

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

The issue does not name files, tests, or entry points. Start by tracing how MySQL migrations currently produce separate ALTER TABLE statements, then compare that path with the requested comma-separated alter_specification syntax; done means compatible operations are emitted as one statement without changing other dialects.

Written by the indexing model from the issue text.

Assessment

Tech stack
mysql, python, sqlalchemy
Domain
databases, tooling
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.