pytest-dev / pytest-dev/pytest-django

Migrations that remove field return column does not exist error when running pytest

Open
#1,045 0 comments 5 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
1.5k
Forks
367
PR merge metrics
No merged PRs in 30d

Description

TLDR of error:
Add a migration that removes a field using the django makemigrations flow. Running migrations using django is fine, no errors returned.
Running pipenv run pytest sample/tests.py returns the error ERROR sample/tests.py::TestModelTestCase::test_example - django.db.utils.ProgrammingError: column sample_testmodel.original_text does not exist.

I run into an error when testing a django application with pytest, more specifically when running tests if I have done a migration that removes a field used in an earlier migration. Doing so outside of tests gives no error.

To reproduce:

models.py
from django.db import models

class TestModel(models.Model):
    # We started with this field, but now longer need it
    # original_text = models.CharField(max_length=255)
    # We added this field and then moved over to it
    new_text = models.CharField(max_length=511)
migrations/0001_initial.py
from django.db import migrations, models


class Migration(migrations.Migration):

    initial = True

    dependencies = [
    ]

    operations = [
        migrations.CreateModel(
            name='TestModel',
            fields=[
                ('id', models.BigAutoField(auto_created=True, primary_key=True, serialize=False, verbose_name='ID')),
                ('original_text', models.CharField(max_length=255)),
            ],
        ),
    ]
migrations/0002_testmodel_new_text.py
from django.db import migrations, models


def move_text_to_other_field_and_append_new(apps, schema_editor):
    TestModel = apps.get_model("sample", "TestModel")

    for test_model in TestModel.objects.all():
        test_model.new_text = test_model.original_text + " new"
        test_model.save()

class Migration(migrations.Migration):

    dependencies = [
        ('sample', '0001_initial'),
    ]

    operations = [
        migrations.AddField(
            model_name='testmodel',
            name='new_text',
            field=models.CharField(default='', max_length=511),
            preserve_default=False,
        ),
        migrations.RunPython(move_text_to_other_field_and_append_new, migrations.RunPython.noop)
    ]
migrations/0003_remove_testmodel_original_text.py
from django.db import migrations


class Migration(migrations.Migration):

    dependencies = [
        ('sample', '0002_testmodel_new_text'),
    ]

    operations = [
        migrations.RemoveField(
            model_name='testmodel',
            name='original_text',
        ),
    ]
tests.py
from django.test import TestCase


class TestModelTestCase(TestCase):

    def test_example(self):
        self.assertEqual(1, 1)

This is a minimal setup to reproduce the error. Now when running pipenv run pytest sample/tests.py I get the error ERROR sample/tests.py::TestModelTestCase::test_example - django.db.utils.ProgrammingError: column sample_testmodel.original_text does not exist. In the error stack it clearly states the following:

sample/migrations/0002_testmodel_new_text.py:9: in move_text_to_other_field_and_append_new
    for test_model in TestModel.objects.all():

And then:

self = <django.db.backends.utils.CursorWrapper object at 0x7f3e53a40e80>, sql = 'SELECT "sample_testmodel"."id", "sample_testmodel"."original_text", "sample_testmodel"."new_text" FROM "sample_testmodel"', params = ()
ignored_wrapper_args = (False, {'connection': <django.db.backends.postgresql.base.DatabaseWrapper object at 0x7f3e5d7e5d90>, 'cursor': <django.db.backends.utils.CursorWrapper object at 0x7f3e53a40e80>})

    def _execute(self, sql, params, *ignored_wrapper_args):
        self.db.validate_no_broken_transaction()
        with self.db.wrap_database_errors:
            if params is None:
                # params default might be backend specific.
                return self.cursor.execute(sql)
            else:
>               return self.cursor.execute(sql, params)
E               django.db.utils.ProgrammingError: column sample_testmodel.original_text does not exist
E               LINE 1: SELECT "sample_testmodel"."id", "sample_testmodel"."original...
E                                                       ^

../../../.local/share/virtualenvs/sprancher-wR5ufyMh/lib/python3.8/site-packages/django/db/backends/utils.py:84: ProgrammingError

This has been bugging me for a long time. It only goes wrong with the tests. Commenting out the RunPython line in 0002 has worked, but is obviously not a long term solution. Running the tests before adding 0003 outputs green lights, no issues up to that point. The order of execution seems incorrect to me or something, but I can't really wrap my head around it.

My question is, why does this happen and how do I prevent this from happening?

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by running pipenv run pytest sample/tests.py with the migration sequence in migrations/0001_initial.py, migrations/0002_testmodel_new_text.py, and migrations/0003_remove_testmodel_original_text.py. Inspect the failure at move_text_to_other_field_and_append_new and compare test database migration behavior with normal Django migrations. Done means the reproduction no longer raises the missing-column error while preserving the data-migration behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
django, python
Domain
backend, testing-qa
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.