Incompatible scopes of live_server and transactional_db

The live_server fixture is session-scoped while the transactional_db fixture (which live_server implicitly enables) is function-scoped.

As a consequence, live_server can handle requests during the whole duration of the test suite. These requests can collide with the database flushes that transactional_db performs after each test or with the periods when access to the database is disabled between two tests.

Here's what the sequence of events looks like:

• Session: live_server fixture starts a LiveServerThread
  • Function: transactional_db fixture runs (triggered by the function-scoped, autouse _live_server_helper fixture) and adds TransactionTestCase._post_teardown to finalizers
  • Function: test makes a request to the live server, perhaps from a selenium-driven browser, and terminates before the live server can respond to that request (this can happen for various reasons; a common one is accessing a page that makes AJAX requests when it loads towards the end of the test and exiting the test function before all these requests have completed)
  • Function: transactional_db finalizer runs and attempts to flush the database, conflicting with the requests the live server is still processing
• Session: live_server finalizer waits for LiveServerThread to terminate

The conflict can cause deadlocks between any SQL query from a HTTP request handled by the live server and the query that flushes the database, which looks like this:

Exception Database test_xxxxxxxx couldn't be flushed. Possible reasons:
  * The database isn't running or isn't configured correctly.
  * At least one of the expected database tables doesn't exist.
  * The SQL was invalid.
Hint: Look at the output of 'django-admin sqlflush'. That's the SQL this command wasn't able to run.
The full error: deadlock detected
DETAIL:  Process 37253 waits for AccessExclusiveLock on relation 131803 of database 131722; blocked by process 37250.
Process 37250 waits for AccessShareLock on relation 132659 of database 131722; blocked by process 37253.
HINT:  See server log for query details.

This failure mode is particularly annoying because the database isn't flushed, requiring to next test run to re-create the database and thus negating the benefits of --reuse-db.

If database reuse isn't enabled, destroying the test database can fail with:

django.db.utils.OperationalError: database "test_xxxxxxxx" is being accessed by other users
DETAIL:  There is 1 other session using the database.

I've also seen SQL queries from HTTP requests handled by the live server to fail with:

Failed: Database access not allowed, use the "django_db" mark, or the "db" or "transactional_db" fixtures to enable it.

because the transactional_db finalizer has flushed the database and blocked access until the next test.

I'm planning to look for workarounds and will update this ticket with my findings.

Trace the live_server and transactional_db fixtures, including the function-scoped _live_server_helper, LiveServerThread, and TransactionTestCase._post_teardown finalizer. Reproduce the failure with the described request and teardown sequence, and verify that live-server requests no longer overlap database flushing or disabled database access.