Occasionally seeing 'Cannot promote terminal deployment' for blue/green approach and deployment being automatically promoted

If you have a question, prepend your issue with [question] or preferably use the nomad mailing list.

If filing a bug please include the following:

Nomad version

Output from nomad version
0.8.4

Operating system and Environment details

Ubuntu 16.04
Running a 5 node cluster on Ec2 t2.medium which each runs vault, nomad, and consul.

Issue

We use canary deployment strategy with matching canaries count to achieve blue green deployment.
Our strategy is to poll for the deployment status until all canary allocs are healthy and then promote (we fail if any one becomes unhealthy by timeout). We use the docker driver.

However,
we occasinally encounter an error when calling promote
Cannot promote terminal deployment

Which is clear that the deployment is terminal, but I neither know what that means or how a canary deployment can become terminal without my doing. The odd thing is the container is the correct build version. (Build version is passed via env args to container and served in API)

Reproduction steps

TLDR;
Interpolate docker image name and build id into job spec
convert job ACL -> json
Write to jobs endpoint
Read deployment id
Poll deployment for health
Promote deployment

Nomad Server logs (if appropriate)

Nomad Client logs (if appropriate)

Job file (if appropriate)

job "core-api-unstable" {
  datacenters = ["dc1"]

  type = "service"

  group "core-api-unstable" {

    count = 2

    update {
      # https://www.nomadproject.io/docs/job-specification/update.html
      max_parallel     = 1
      canary           = 2
      min_healthy_time = "10s"
      healthy_deadline = "1m"
      auto_revert      = true
      health_check = "checks"
      progress_deadline = "2m"
    }

    task "core-api-unstable" {

      template {
        data = <<EOH

...REDACTED

EOH
        destination = "secrets/file.env"
        env = true
      }

      logs {
        max_files = 10
        max_file_size = 10
      }
      driver = "docker"
      config {
        image = "$[[NOMAD_META_DOCKER_IMAGE_URI]]"
        network_mode = "host"
      }
      resources {
        cpu    = 500 # MHz
        memory = 500 # Megabytes
        network {
          mbits = 1
          port "http" {}
        }
      }
      # tells nomad how to register service with consul
      service {
        name = "core-api-unstable"

        # this tag is used by fabio to know that any requests with header Host: 'unstable-core-api.joe.coffee'
        # should target this job group
        tags = ["REDACTED/"]


        canary_tags = ["REDACTED"]
        port = "http"
        check {
          path = "/api/v1"
          name     = "unstable-core-api-alive"
          type     = "http"
          interval = "5s"
          timeout  = "2s"
        }
      }

      # TODO - extract these from vault
      env {
        "PG_DB_HOST" = "REDACTED"
        "PG_DB_PORT" = "5432"
        # dynamic ports automatically exposed via this env
        # format NOMAD_PORT_${portname}
        "PORT" = "${NOMAD_PORT_http}"
        "NODE_ENV" = "unstable"
        "BUILD_ID" = "$[[NOMAD_META_BUILD_ID]]"
      }
    }
  }
}

No source file or test is identified. Start with the deployment lifecycle for the supplied Nomad 0.8.4 job, especially the update stanza and the jobs-endpoint, polling, and promote sequence; reproduce the canary transition and determine why it becomes terminal before promotion. Done means a valid blue/green promotion does not intermittently return “Cannot promote terminal deployment,” with coverage for the observed transition.