A couple features to facilitate a branch-by-default workflow

Preface

My team recently made the switch to Alembic, and it's been great! However, it took some jimmy-rigging to set up the workflow we need. I imagine there must be other teams that would benefit from a similar setup, so I thought I'd post here describing (1) what I came up with, and (2) a few minor(?) features that would make it even nicer. Of course, I realize Alembic can't cater to every possible workflow, but no harm in asking right ;). I would be interested in implementing said features, if there is interest. Alternatively -- critique my approach! Maybe I missed something which renders these features unnecessary.

Background

I work on a team with five other developers and, due to the nature of the project, database revisions are common: we add about three per week (collectively), sometimes more. We have three environments in our normal flow -- qa, stage, production -- plus a sandbox which doesn't get much use. We release to production about twice a month.

The problem

Our previous migration tool did not support branching, and trying to force a strictly linear migration history was proving to be a nightmare, especially in regards to moving code between environments. This was especially annoying considering that most of the time, the order in which revisions from the same sprint are applied doesn't actually matter! ... but the tool required a linear history. What we really want is a "semi-ordered" migration history, where revisions from 6 months ago will certainly precede revisions from this month, but on a smaller time-scale (say, a sprint) migrations are unordered by default. (By default is key -- sometimes we do need to specify a dependency between migrations from the same sprint.)

The solution

So that's what we were after, and Alembic's branching got us 90% there, but a little more is needed for a complete, developer-friendly solution. There were two points to still address:

• when do merges happen? we can't just branch outwards forever...
• when creating a new revision, from which existing revision does the developer branch off of? and how do they do it easily, in a manner not prone to errors?

When do merges happen?

Before each release to production, on the release branch, add a revision merging all heads, and give it a special message. We call it a "release revision", and use this script:

# create_release_revision.sh

if [ -z "$1" ]; then
    echo 'Please provide the release version number. Example:'
    echo '$ create_release_revision.sh 1.29.0'
  exit 1
fi

alembic merge heads --message "[RELEASE-MERGE] v$1"

After the release, we merge production back into qa and stage so that they get the changes added in the release branch (version number, changelog, and now the release revision too).

Creating a new revision...

When a developer needs to create a new revision, and its only dependencies are already in production (as is usually the case), they are to branch off of the most recent "release revision", as generated by the script above. To make this easy, we use the following script:

# create_revision_branch.py

import sys
import os

# don't import from current directory because alembic/ gets in the way
sys.path.remove(os.getcwd())

from alembic.config import Config           # noqa
from alembic.config import CommandLine      # noqa
from alembic.script import ScriptDirectory  # noqa


def get_last_release_rev_id():
    script_dir = ScriptDirectory('alembic')
    for script in script_dir.walk_revisions():
        if script.doc.startswith('[RELEASE-MERGE]'):
            return script.revision
    raise Exception('No release revision found in history')


def main():
    cmd_line = CommandLine()

    # drop script name from argv
    # argparse does this by default iff argv is not provided
    argv = sys.argv[1:]

    # play the roll of `alembic revision`
    argv.insert(0, 'revision')

    options = cmd_line.parser.parse_args(argv)

    options.head = get_last_release_rev_id()
    options.splice = True

    cfg = Config(
        file_=options.config,
        ini_section=options.name,
        cmd_opts=options,
    )
    cmd_line.run_cmd(cfg, options)


if __name__ == '__main__':
    main()

This script acts just like alembic revision, except it automatically sets the --head flag to the most recent release revision, and also passes --splice. If the new migration depends on another migration which is not yet released, the developer must instead uses bare alembic revision and set --head themselves.

---

So that's what we've got, and we're so glad to finally have a system that works, technically speaking, even if it's just a little bit clunky. Now, if story A is merged into QA before story B, B can still be merged into stage before A! What a concept!

Feature proposal

That said, ideally we would not need to use a script for an operation as basic as creating a new revision, and I believe it would be fairly straightforward to give this workflow proper support in a non-invasive manner. It would look something like this:

• a setting in alembic.cfg branch_by_default perhaps, which would default to False, but when True, would make alembic revision branch off of the most recent “release revision”, unless --head is specified, in which case it would work as it does now
• a new argument to alembic merge which would somehow designate the revision a "release revision" (as I've been calling it in this post). It would maybe be called --release, but it would be nice to think of something more general, since these revisions don't necessarily need to correspond with releases.

The main question here is of course how to designate release revisions. Would it just be a message prefix, as in the scripts above, or is there a better way? I've seen something about "tags" when browsing the docs, but haven't made sense of them, and come to think of it, I'm not even sure they apply to revisions, but if they do, that sounds like it could be a candidate?

Start with the existing create_revision_branch.py and create_release_revision.sh examples, then trace the alembic revision and alembic merge entry points. Determine the intended configuration and CLI semantics, including the --head override and how a release revision is designated; done means the workflow is specified clearly enough to implement and verify.