Auto-migrate considered harmful
- Dominant language
- No language data
- Stars
- 39
- Forks
- 1
- PR merge metrics
- No merged PRs in 30d
Description
CKAN extensions are often written to create and alter their database tables automatically on run-time. The idea is that when you install or upgrade an extension that uses tables, then you don't have to remember to run a paster command to do the migration - it just happens automatically when you run any paster command or restart the server.
This is seen in:
- ckanext-harvest - https://github.com/ckan/ckanext-harvest/blob/master/ckanext/harvest/model/__init__.py#L43
- ckanext-spatial - https://github.com/ckan/ckanext-spatial/blob/master/ckanext/spatial/model/package_extent.py#L19
- ckanext-showcase - https://github.com/ckan/ckanext-showcase/blob/master/ckanext/showcase/model/__init__.py#L18
- ckanext-issues - https://github.com/okfn/ckanext-issues/blob/master/ckanext/issues/model/__init__.py#L30
etc
However data.gov.uk has frequently run into problems with this approach installing them on a live server. When the extension is installed and you restart apache, then the multiple CKAN processes will all try to get a write lock on postgres and then proceed to all create or alter the tables. You either get dead-lock or lots of errors. I've ended up with minutes of downtime, aborted upgrades and fixing table migrations manually while the server remains down. It's particularly bad for ckanext-spatial as it had migration during some upgrades of the code. The only way to upgrade it safely is to stop apache, upgrade code, run an unrelated paster command that does the upgrade and then start apache again.
I think it would be better if migrations were done using a specific paster command, as I have done in ckanext-archiver (`paster --plugin=ckanext-archiver archiver init`). It means there are 5 steps to the install instead of 4. It is easier to understand than running an unrelated paster command. And you don't have down-time - unplanned or planned.
https://github.com/datagovuk/ckanext-archiver
Can we agree that this is a better way to do it?
Contributor guide
No contributing guide indexed for this repository
Research direction
Read the linked model files in ckanext-harvest, ckanext-spatial, ckanext-showcase, and ckanext-issues, then compare them with ckanext-archiver's `archiver init` command. The issue does not identify a target repository, implementation files, tests, or an agreed definition of done; the immediate outcome would be a project decision about migration policy.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- postgres
- Domain
- databases
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 20/100