sqlalchemy / sqlalchemy/alembic
ENUM with metadata is not translated to sql CREATE TYPE in autogenerate
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 4.4k
- Forks
- 375
- PR merge metrics
- No merged PRs in 30d
Description
Hi,
I know that ENUM autogeneration is not entirely complete and polished, but I think I might have encountered not something not working, but working incorrectly :)
Describe the bug
- Autogenerate for postgres ENUM with
metadata=...generates a migration withMetadata(bind=None)without importingMetadata, resulting in NameError when running migration. - The enum type is not created in sql if specified with
metadata=..., even though it seems like a well-documented use-case in sqlalchemy docs.
Expected behavior
from sqlalchemy import MetaData- Create the enum type in the database
To Reproduce
from enum import Enum
from sqlalchemy import BigInteger, Column, MetaData, Table
from sqlalchemy.dialects.postgresql import ENUM
metadata = MetaData(schema="myschema")
class SomeEnum(str, Enum):
OPEN = "OPEN"
CLOSED = "CLOSED"
some_enum = ENUM(
SomeEnum,
metadata=metadata,
schema=metadata.schema,
)
some_table = Table(
"some_table",
metadata,
Column("id", BigInteger, primary_key=True),
Column("some_enum_value", some_enum),
)
Resulting migration file:
"""initial
Revision ID: 653c623403f4
Revises:
Create Date: 2021-06-24 18:15:52.750889
"""
from alembic import op
import sqlalchemy as sa
from sqlalchemy.dialects import postgresql
# revision identifiers, used by Alembic.
revision = '653c623403f4'
down_revision = None
branch_labels = None
depends_on = None
def upgrade():
# ### commands auto generated by Alembic - please adjust! ###
op.create_table('some_table',
sa.Column('id', sa.BigInteger(), nullable=False),
sa.Column('some_enum_value', postgresql.ENUM('OPEN', 'CLOSED', name='someenum', schema='myschema', metadata=MetaData(bind=None)), nullable=True),
sa.PrimaryKeyConstraint('id'),
schema='myschema'
)
# ### end Alembic commands ###
def downgrade():
# ### commands auto generated by Alembic - please adjust! ###
op.drop_table('some_table', schema='myschema')
# ### end Alembic commands ###
Error
18:15 $ alembic upgrade head --sql
INFO [alembic.runtime.migration] Context impl PostgresqlImpl.
INFO [alembic.runtime.migration] Generating static SQL
INFO [alembic.runtime.migration] Will assume transactional DDL.
BEGIN;
CREATE TABLE myschema.alembic_version (
version_num VARCHAR(32) NOT NULL,
CONSTRAINT alembic_version_pkc PRIMARY KEY (version_num)
);
INFO [alembic.runtime.migration] Running upgrade -> 653c623403f4, initial
-- Running upgrade -> 653c623403f4
Traceback (most recent call last):
File "/home/.../bin/alembic", line 8, in <module>
sys.exit(main())
File "/home/.../lib/python3.8/site-packages/alembic/config.py", line 559, in main
CommandLine(prog=prog).main(argv=argv)
File "/home/.../lib/python3.8/site-packages/alembic/config.py", line 553, in main
self.run_cmd(cfg, options)
File "/home/.../lib/python3.8/site-packages/alembic/config.py", line 530, in run_cmd
fn(
File "/home/.../lib/python3.8/site-packages/alembic/command.py", line 293, in upgrade
script.run_env()
File "/home/.../lib/python3.8/site-packages/alembic/script/base.py", line 490, in run_env
util.load_python_file(self.dir, "env.py")
File "/home/.../lib/python3.8/site-packages/alembic/util/pyfiles.py", line 97, in load_python_file
module = load_module_py(module_id, path)
File "/home/.../lib/python3.8/site-packages/alembic/util/compat.py", line 184, in load_module_py
spec.loader.exec_module(module)
File "<frozen importlib._bootstrap_external>", line 783, in exec_module
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "migrations/env.py", line 90, in <module>
run_migrations_offline()
File "migrations/env.py", line 60, in run_migrations_offline
context.run_migrations()
File "<string>", line 8, in run_migrations
File "/home/.../lib/python3.8/site-packages/alembic/runtime/environment.py", line 813, in run_migrations
self.get_context().run_migrations(**kw)
File "/home/.../lib/python3.8/site-packages/alembic/runtime/migration.py", line 561, in run_migrations
step.migration_fn(**kw)
File "/home/.../migrations/versions/2021_06_24_653c623403f4_initial.py", line 23, in upgrade
sa.Column('some_enum_value', postgresql.ENUM('OPEN', 'CLOSED', name='someenum', schema='myschema', metadata=MetaData(bind=None)), nullable=True),
NameError: name 'MetaData' is not defined
Note that adding the missing import (from sqlalchemy import MetaData) results in a proper sql, but without the CREATE TYPE:
alembic upgrade head --sql
INFO [alembic.runtime.migration] Context impl PostgresqlImpl.
INFO [alembic.runtime.migration] Generating static SQL
INFO [alembic.runtime.migration] Will assume transactional DDL.
BEGIN;
CREATE TABLE myschema.alembic_version (
version_num VARCHAR(32) NOT NULL,
CONSTRAINT alembic_version_pkc PRIMARY KEY (version_num)
);
INFO [alembic.runtime.migration] Running upgrade -> 653c623403f4, initial
-- Running upgrade -> 653c623403f4
CREATE TABLE myschema.some_table (
id BIGSERIAL NOT NULL,
some_enum_value myschema.someenum,
PRIMARY KEY (id)
);
INSERT INTO myschema.alembic_version (version_num) VALUES ('653c623403f4');
COMMIT;
Versions.
- OS: Ubuntu 20.04.2
- Python: 3.8.5
- Alembic: 1.6.5
- SQLAlchemy: 1.3.24 (I can't upgrade to 1.4 yet)
- Database: PostgreSQL 12.7
- DBAPI: psycopg2-binary 2.8.6
Additional context
Note that removing the metadata argument makes the issue disappear:
some_enum = ENUM(
SomeEnum,
# metadata=metadata,
schema=metadata.schema,
)
gives no metadata=Metadata(bind=None), so no NameError is given as a result:
sa.Column('some_enum_value', postgresql.ENUM('OPEN', 'CLOSED', name='someenum', schema='myschema'), nullable=True),
Also, the resulting SQL is as expected:
-- Running upgrade -> c2d662ec840a
CREATE TYPE myschema.someenum AS ENUM ('OPEN', 'CLOSED');
CREATE TABLE myschema.some_table (
id BIGSERIAL NOT NULL,
some_enum_value myschema.someenum,
PRIMARY KEY (id)
);
According to the sqlalchemy docs:
To use a common enumerated type between multiple tables, the best practice is to declare the Enum or ENUM independently, and associate it with the MetaData object itself:
...
If we specify checkfirst=True, the individual table-level create operation will check for the ENUM and create if not exists:
So I guess in scenario where I provided the metadata argument, it is expected that the enum would not be created by sqlalchemy table create unless asked to with checkfirst=True.
However I think that alembic should either produce the same SQL statements in both cases (metadata argument to enum specified or not), or at least inform in the docs how to enforce the checkfirst-like behaviour.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Reproduce the issue with the provided MetaData, PostgreSQL ENUM, and generated migration, then inspect Alembic's autogenerate handling of the resulting migration. Verify the generated migration can run without a missing MetaData import and that offline SQL includes CREATE TYPE before CREATE TABLE.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- postgresql, python, sqlalchemy
- Domain
- databases, tooling
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100