hashicorp / hashicorp/nomad

Job versions from failed canary deployments are re-allocated during migration

Open
#5,619 0 comments 0 reactions 0 assignees View on GitHub
stage/needs-verification theme/deployments type/bug
Dominant language
Go
Stars
17k
Forks
2.1k
Avg merge
1d 9h
Merged PRs (30d)
105

Description

### Nomad version
`Nomad v0.8.6`

### Operating system and Environment details
`Ubuntu 16.04.3 LTS`

### Issue
When the last deployment for a job with a canary is marked as failed _and_ an allocation from that job is then migrated, Nomad allocates the failed version rather than the last successful version of the job. This could be my own misconception of the failed deployment state, but when a deployment is failed it should in some way invalidate the job so that it cannot be automatically re-allocated during migration.

### Reproduction steps
1. run a job with canary enabled
1. wait for it to successfully deploy
1. update the job and run it again
1. fail the deployment
1. drain any node that contains a running allocation of the job
1. wait for the allocation to migrate
1. verify the job version of the migrated allocation

### Expected results
The new allocation should match those of the last successful deployment.

### Actual results
The new allocation matches those of the last deployment, even if it was marked as failed.

### Job files
"passing" version:
```
job "migration-test" {
region = "us-east-1"
datacenters = ["us-east-1a", "us-east-1b", "us-east-1d"]

constraint {
attribute = "${node.class}"
value = "staging"
}

update {
max_parallel = 1
health_check = "checks"
stagger = "120s"
canary = 1
healthy_deadline = "5m"
}

group "dummy" {
count = 1

restart {
interval = "1m"
attempts = 2
delay = "15s"
mode = "delay"
}

task "dummy" {
driver = "docker"

config {
image = "alpine:latest"
command = "/bin/sh"
args = ["-c", "echo pass && /usr/bin/tail -f /dev/null"]
}

resources {
cpu = 100
memory = 32

network {
mbits = 10
}
}
}
}
}
```

"failing" version (trivially echo `fail` instead of `pass`)
```
job "migration-test" {
region = "us-east-1"
datacenters = ["us-east-1a", "us-east-1b", "us-east-1d"]

constraint {
attribute = "${node.class}"
value = "staging"
}

update {
max_parallel = 1
health_check = "checks"
stagger = "120s"
canary = 1
healthy_deadline = "5m"
}

group "dummy" {
count = 1

restart {
interval = "1m"
attempts = 2
delay = "15s"
mode = "delay"
}

task "dummy" {
driver = "docker"

config {
image = "alpine:latest"
command = "/bin/sh"
args = ["-c", "echo fail && /usr/bin/tail -f /dev/null"]
}

resources {
cpu = 100
memory = 32

network {
mbits = 10
}
}
}
}
}
```

### Logs
```
timestamp event_type message
April 26th 2019, 19:56:22.756 job {"Stop":false,"Region":"us-east-1","Namespace":"default","ID":"migration-test","ParentID":"","Name":"migration-test","Type":"service","Priority":50,"AllAtOnce":false,"Datacenters":["us-east-1a","us-east-1b","us-east-1d"],"Constraints":[{"LTarget":"${node.class}","RTarget":"staging","Operand":"="}],"TaskGroups":[{"Name":"dummy","Count":1,"Constraints":null,"Tasks":[{"Name":"dummy","Driver":"docker","User":"","Config":{"args":["-c","echo pass \u0026\u0026 /usr/bin/tail -f /dev/null"],"command":"/bin/sh","image":"alpine:latest"},"Constraints":null,"Env":null,"Services":null,"Resources":{"CPU":100,"MemoryMB":32,"DiskMB":0,"IOPS":0,"Networks":[{"Device":"","CIDR":"","IP":"","MBits":10,"ReservedPorts":null,"DynamicPorts":null}]},"Meta":null,"KillTimeout":5000000000,"LogConfig":{"MaxFiles":10,"MaxFileSizeMB":10},"Artifacts":null,"Vault":null,"Templates":null,"DispatchPayload":null,"Leader":false,"ShutdownDelay":0,"KillSignal":""}],"RestartPolicy":{"Interval":60000000000,"Attempts":2,"Delay":15000000000,"Mode":"delay"},"EphemeralDisk":{"Sticky":false,"Migrate":false,"SizeMB":300},"Update":{"Stagger":120000000000,"MaxParallel":1,"HealthCheck":"checks","MinHealthyTime":10000000000,"HealthyDeadline":300000000000,"AutoRevert":false,"Canary":1},"Meta":null}],"Update":{"Stagger":120000000000,"MaxParallel":1,"HealthCheck":"","MinHealthyTime":0,"HealthyDeadline":0,"AutoRevert":false,"Canary":0},"Periodic":null,"ParameterizedJob":null,"Payload":null,"Meta":null,"VaultToken":"","Status":"pending","StatusDescription":"","Stable":false,"Version":0,"SubmitTime":1556308579423112620,"CreateIndex":5240326,"ModifyIndex":5240326,"JobModifyIndex":5240326}
April 26th 2019, 19:56:22.756 job {"Stop":false,"Region":"us-east-1","Namespace":"default","ID":"migration-test","ParentID":"","Name":"migration-test","Type":"service","Priority":50,"AllAtOnce":false,"Datacenters":["us-east-1a","us-east-1b","us-east-1d"],"Constraints":[{"LTarget":"${node.class}","RTarget":"staging","Operand":"="}],"TaskGroups":[{"Name":"dummy","Count":1,"Constraints":null,"Tasks":[{"Name":"dummy","Driver":"docker","User":"","Config":{"args":["-c","echo pass \u0026\u0026 /usr/bin/tail -f /dev/null"],"command":"/bin/sh","image":"alpine:latest"},"Constraints":null,"Env":null,"Services":null,"Resources":{"CPU":100,"MemoryMB":32,"DiskMB":0,"IOPS":0,"Networks":[{"Device":"","CIDR":"","IP":"","MBits":10,"ReservedPorts":null,"DynamicPorts":null}]},"Meta":null,"KillTimeout":5000000000,"LogConfig":{"MaxFiles":10,"MaxFileSizeMB":10},"Artifacts":null,"Vault":null,"Templates":null,"DispatchPayload":null,"Leader":false,"ShutdownDelay":0,"KillSignal":""}],"RestartPolicy":{"Interval":60000000000,"Attempts":2,"Delay":15000000000,"Mode":"delay"},"EphemeralDisk":{"Sticky":false,"Migrate":false,"SizeMB":300},"Update":{"Stagger":120000000000,"MaxParallel":1,"HealthCheck":"checks","MinHealthyTime":10000000000,"HealthyDeadline":300000000000,"AutoRevert":false,"Canary":1},"Meta":null}],"Update":{"Stagger":120000000000,"MaxParallel":1,"HealthCheck":"","MinHealthyTime":0,"HealthyDeadline":0,"AutoRevert":false,"Canary":0},"Periodic":null,"ParameterizedJob":null,"Payload":null,"Meta":null,"VaultToken":"","Status":"running","StatusDescription":"","Stable":false,"Version":0,"SubmitTime":1556308579423112620,"CreateIndex":5240326,"ModifyIndex":5240328,"JobModifyIndex":5240326}
April 26th 2019, 19:56:27.727 deployment {"ID":"242a0914-90fd-0c9c-ac5d-36a9f83ef198","Namespace":"default","JobID":"migration-test","JobVersion":0,"JobModifyIndex":5240326,"JobCreateIndex":5240326,"TaskGroups":{"dummy":{"PlacedCanaries":null,"AutoRevert":false,"Promoted":false,"DesiredCanaries":0,"DesiredTotal":1,"PlacedAllocs":1,"HealthyAllocs":0,"UnhealthyAllocs":0}},"Status":"running","StatusDescription":"Deployment is running","CreateIndex":5240328,"ModifyIndex":5240328}
April 26th 2019, 19:56:28.743 allocation {"Name":"migration-test.dummy[0]","NodeID":"6c16340a-24fe-544d-a8b4-4f91fe200a8c","AllocationID":"a92f8ba6-c9cd-5789-9cb6-faa00a785eb4","DesiredStatus":"run","DesiredDescription":"","ClientStatus":"pending","ClientDescription":"","JobID":"migration-test","GroupName":"dummy","TaskName":"dummy","EvalID":"b2154fa3-3d30-9b2d-bd6e-5bd99f9f8c2d","TaskState":"pending","TaskFailed":false,"TaskStartedAt":"0001-01-01T00:00:00Z","TaskFinishedAt":"0001-01-01T00:00:00Z","TaskEvent":{"Type":"Received","Time":1556308579599265636,"DisplayMessage":"Task received by client","Details":{},"FailsTask":false,"RestartReason":"","SetupError":"","DriverError":"","DriverMessage":"","ExitCode":0,"Signal":0,"Message":"","KillReason":"","KillTimeout":0,"KillError":"","StartDelay":0,"DownloadError":"","ValidationError":"","DiskLimit":0,"DiskSize":0,"FailedSibling":"","VaultError":"","TaskSignalReason":"","TaskSignal":"","GenericSource":""}}
April 26th 2019, 19:56:28.743 allocation {"Name":"migration-test.dummy[0]","NodeID":"6c16340a-24fe-544d-a8b4-4f91fe200a8c","AllocationID":"a92f8ba6-c9cd-5789-9cb6-faa00a785eb4","DesiredStatus":"run","DesiredDescription":"","ClientStatus":"pending","ClientDescription":"","JobID":"migration-test","GroupName":"dummy","TaskName":"dummy","EvalID":"b2154fa3-3d30-9b2d-bd6e-5bd99f9f8c2d","TaskState":"pending","TaskFailed":false,"TaskStartedAt":"0001-01-01T00:00:00Z","TaskFinishedAt":"0001-01-01T00:00:00Z","TaskEvent":{"Type":"Driver","Time":1556308579615479698,"DisplayMessage":"Downloading image alpine:latest","Details":{"driver_message":"Downloading image alpine:latest"},"FailsTask":false,"RestartReason":"","SetupError":"","DriverError":"","DriverMessage":"Downloading image alpine:latest","ExitCode":0,"Signal":0,"Message":"","KillReason":"","KillTimeout":0,"KillError":"","StartDelay":0,"DownloadError":"","ValidationError":"","DiskLimit":0,"DiskSize":0,"FailedSibling":"","VaultError":"","TaskSignalReason":"","TaskSignal":"","GenericSource":""}}
April 26th 2019, 19:56:28.743 allocation {"Name":"migration-test.dummy[0]","NodeID":"6c16340a-24fe-544d-a8b4-4f91fe200a8c","AllocationID":"a92f8ba6-c9cd-5789-9cb6-faa00a785eb4","DesiredStatus":"run","DesiredDescription":"","ClientStatus":"running","ClientDescription":"","JobID":"migration-test","GroupName":"dummy","TaskName":"dummy","EvalID":"b2154fa3-3d30-9b2d-bd6e-5bd99f9f8c2d","TaskState":"running","TaskFailed":false,"TaskStartedAt":"2019-04-26T19:56:20.302155035Z","TaskFinishedAt":"0001-01-01T00:00:00Z","TaskEvent":{"Type":"Started","Time":1556308580290594541,"DisplayMessage":"Task started by client","Details":{},"FailsTask":false,"RestartReason":"","SetupError":"","DriverError":"","DriverMessage":"","ExitCode":0,"Signal":0,"Message":"","KillReason":"","KillTimeout":0,"KillError":"","StartDelay":0,"DownloadError":"","ValidationError":"","DiskLimit":0,"DiskSize":0,"FailedSibling":"","VaultError":"","TaskSignalReason":"","TaskSignal":"","GenericSource":""}}
April 26th 2019, 19:56:28.743 allocation {"Name":"migration-test.dummy[0]","NodeID":"6c16340a-24fe-544d-a8b4-4f91fe200a8c","AllocationID":"a92f8ba6-c9cd-5789-9cb6-faa00a785eb4","DesiredStatus":"run","DesiredDescription":"","ClientStatus":"pending","ClientDescription":"","JobID":"migration-test","GroupName":"dummy","TaskName":"dummy","EvalID":"b2154fa3-3d30-9b2d-bd6e-5bd99f9f8c2d","TaskState":"pending","TaskFailed":false,"TaskStartedAt":"0001-01-01T00:00:00Z","TaskFinishedAt":"0001-01-01T00:00:00Z","TaskEvent":{"Type":"Task Setup","Time":1556308579599314323,"DisplayMessage":"Building Task Directory","Details":{"message":"Building Task Directory"},"FailsTask":false,"RestartReason":"","SetupError":"","DriverError":"","DriverMessage":"","ExitCode":0,"Signal":0,"Message":"Building Task Directory","KillReason":"","KillTimeout":0,"KillError":"","StartDelay":0,"DownloadError":"","ValidationError":"","DiskLimit":0,"DiskSize":0,"FailedSibling":"","VaultError":"","TaskSignalReason":"","TaskSignal":"","GenericSource":""}}
April 26th 2019, 19:56:30.728 deployment {"ID":"242a0914-90fd-0c9c-ac5d-36a9f83ef198","Namespace":"default","JobID":"migration-test","JobVersion":0,"JobModifyIndex":5240326,"JobCreateIndex":5240326,"TaskGroups":{"dummy":{"PlacedCanaries":null,"AutoRevert":false,"Promoted":false,"DesiredCanaries":0,"DesiredTotal":1,"PlacedAllocs":1,"HealthyAllocs":1,"UnhealthyAllocs":0}},"Status":"running","StatusDescription":"Deployment is running","CreateIndex":5240328,"ModifyIndex":5240333}
April 26th 2019, 19:56:31.730 deployment {"ID":"242a0914-90fd-0c9c-ac5d-36a9f83ef198","Namespace":"default","JobID":"migration-test","JobVersion":0,"JobModifyIndex":5240326,"JobCreateIndex":5240326,"TaskGroups":{"dummy":{"PlacedCanaries":null,"AutoRevert":false,"Promoted":false,"DesiredCanaries":0,"DesiredTotal":1,"PlacedAllocs":1,"HealthyAllocs":1,"UnhealthyAllocs":0}},"Status":"successful","StatusDescription":"Deployment completed successfully","CreateIndex":5240328,"ModifyIndex":5240335}
April 26th 2019, 19:56:37.688 evaluation {"ID":"b2154fa3-3d30-9b2d-bd6e-5bd99f9f8c2d","Priority":50,"Type":"service","TriggeredBy":"job-register","Namespace":"default","JobID":"migration-test","JobModifyIndex":5240326,"NodeID":"","NodeModifyIndex":0,"DeploymentID":"242a0914-90fd-0c9c-ac5d-36a9f83ef198","Status":"complete","StatusDescription":"","Wait":0,"NextEval":"","PreviousEval":"","BlockedEval":"","FailedTGAllocs":null,"ClassEligibility":null,"EscapedComputedClass":false,"QuotaLimitReached":"","AnnotatePlan":false,"QueuedAllocations":{"dummy":0},"SnapshotIndex":5240327,"CreateIndex":5240327,"ModifyIndex":5240329}
April 26th 2019, 19:56:37.756 job {"Stop":false,"Region":"us-east-1","Namespace":"default","ID":"migration-test","ParentID":"","Name":"migration-test","Type":"service","Priority":50,"AllAtOnce":false,"Datacenters":["us-east-1a","us-east-1b","us-east-1d"],"Constraints":[{"LTarget":"${node.class}","RTarget":"staging","Operand":"="}],"TaskGroups":[{"Name":"dummy","Count":1,"Constraints":null,"Tasks":[{"Name":"dummy","Driver":"docker","User":"","Config":{"args":["-c","echo pass \u0026\u0026 /usr/bin/tail -f /dev/null"],"command":"/bin/sh","image":"alpine:latest"},"Constraints":null,"Env":null,"Services":null,"Resources":{"CPU":100,"MemoryMB":32,"DiskMB":0,"IOPS":0,"Networks":[{"Device":"","CIDR":"","IP":"","MBits":10,"ReservedPorts":null,"DynamicPorts":null}]},"Meta":null,"KillTimeout":5000000000,"LogConfig":{"MaxFiles":10,"MaxFileSizeMB":10},"Artifacts":null,"Vault":null,"Templates":null,"DispatchPayload":null,"Leader":false,"ShutdownDelay":0,"KillSignal":""}],"RestartPolicy":{"Interval":60000000000,"Attempts":2,"Delay":15000000000,"Mode":"delay"},"EphemeralDisk":{"Sticky":false,"Migrate":false,"SizeMB":300},"Update":{"Stagger":120000000000,"MaxParallel":1,"HealthCheck":"checks","MinHealthyTime":10000000000,"HealthyDeadline":300000000000,"AutoRevert":false,"Canary":1},"Meta":null}],"Update":{"Stagger":120000000000,"MaxParallel":1,"HealthCheck":"","MinHealthyTime":0,"HealthyDeadline":0,"AutoRevert":false,"Canary":0},"Periodic":null,"ParameterizedJob":null,"Payload":null,"Meta":null,"VaultToken":"","Status":"running","StatusDescription":"","Stable":true,"Version":0,"SubmitTime":1556308579423112620,"CreateIndex":5240326,"ModifyIndex":5240335,"JobModifyIndex":5240326}
April 26th 2019, 19:57:12.691 evaluation {"ID":"73589b5f-f246-5161-5557-6cbbaf4bcc06","Priority":50,"Type":"service","TriggeredBy":"deployment-watcher","Namespace":"default","JobID":"migration-test","JobModifyIndex":0,"NodeID":"","NodeModifyIndex":0,"DeploymentID":"242a0914-90fd-0c9c-ac5d-36a9f83ef198","Status":"complete","StatusDescription":"","Wait":0,"NextEval":"","PreviousEval":"","BlockedEval":"","FailedTGAllocs":null,"ClassEligibility":null,"EscapedComputedClass":false,"QuotaLimitReached":"","AnnotatePlan":false,"QueuedAllocations":{"dummy":0},"SnapshotIndex":5240334,"CreateIndex":5240334,"ModifyIndex":5240336}
April 26th 2019, 19:57:12.757 job {"Stop":false,"Region":"us-east-1","Namespace":"default","ID":"migration-test","ParentID":"","Name":"migration-test","Type":"service","Priority":50,"AllAtOnce":false,"Datacenters":["us-east-1a","us-east-1b","us-east-1d"],"Constraints":[{"LTarget":"${node.class}","RTarget":"staging","Operand":"="}],"TaskGroups":[{"Name":"dummy","Count":1,"Constraints":null,"Tasks":[{"Name":"dummy","Driver":"docker","User":"","Config":{"args":["-c","echo fail \u0026\u0026 /usr/bin/tail -f /dev/null"],"command":"/bin/sh","image":"alpine:latest"},"Constraints":null,"Env":null,"Services":null,"Resources":{"CPU":100,"MemoryMB":32,"DiskMB":0,"IOPS":0,"Networks":[{"Device":"","CIDR":"","IP":"","MBits":10,"ReservedPorts":null,"DynamicPorts":null}]},"Meta":null,"KillTimeout":5000000000,"LogConfig":{"MaxFiles":10,"MaxFileSizeMB":10},"Artifacts":null,"Vault":null,"Templates":null,"DispatchPayload":null,"Leader":false,"ShutdownDelay":0,"KillSignal":""}],"RestartPolicy":{"Interval":60000000000,"Attempts":2,"Delay":15000000000,"Mode":"delay"},"EphemeralDisk":{"Sticky":false,"Migrate":false,"SizeMB":300},"Update":{"Stagger":120000000000,"MaxParallel":1,"HealthCheck":"checks","MinHealthyTime":10000000000,"HealthyDeadline":300000000000,"AutoRevert":false,"Canary":1},"Meta":null}],"Update":{"Stagger":120000000000,"MaxParallel":1,"HealthCheck":"","MinHealthyTime":0,"HealthyDeadline":0,"AutoRevert":false,"Canary":0},"Periodic":null,"ParameterizedJob":null,"Payload":null,"Meta":null,"VaultToken":"","Status":"running","StatusDescription":"","Stable":false,"Version":1,"SubmitTime":1556308629274422003,"CreateIndex":5240326,"ModifyIndex":5240337,"JobModifyIndex":5240337}
April 26th 2019, 19:57:13.746 allocation {"Name":"migration-test.dummy[0]","NodeID":"95c779e9-2039-0b22-7965-45ddaff952f6","AllocationID":"d1e64eab-3fdf-e630-2d4f-60365e2eb67a","DesiredStatus":"run","DesiredDescription":"","ClientStatus":"pending","ClientDescription":"","JobID":"migration-test","GroupName":"dummy","TaskName":"dummy","EvalID":"c9c1892a-a16f-cea2-955a-c85052ed0683","TaskState":"pending","TaskFailed":false,"TaskStartedAt":"0001-01-01T00:00:00Z","TaskFinishedAt":"0001-01-01T00:00:00Z","TaskEvent":{"Type":"Task Setup","Time":1556308629416874165,"DisplayMessage":"Building Task Directory","Details":{"message":"Building Task Directory"},"FailsTask":false,"RestartReason":"","SetupError":"","DriverError":"","DriverMessage":"","ExitCode":0,"Signal":0,"Message":"Building Task Directory","KillReason":"","KillTimeout":0,"KillError":"","StartDelay":0,"DownloadError":"","ValidationError":"","DiskLimit":0,"DiskSize":0,"FailedSibling":"","VaultError":"","TaskSignalReason":"","TaskSignal":"","GenericSource":""}}
April 26th 2019, 19:57:13.746 allocation {"Name":"migration-test.dummy[0]","NodeID":"95c779e9-2039-0b22-7965-45ddaff952f6","AllocationID":"d1e64eab-3fdf-e630-2d4f-60365e2eb67a","DesiredStatus":"run","DesiredDescription":"","ClientStatus":"pending","ClientDescription":"","JobID":"migration-test","GroupName":"dummy","TaskName":"dummy","EvalID":"c9c1892a-a16f-cea2-955a-c85052ed0683","TaskState":"pending","TaskFailed":false,"TaskStartedAt":"0001-01-01T00:00:00Z","TaskFinishedAt":"0001-01-01T00:00:00Z","TaskEvent":{"Type":"Received","Time":1556308629416825065,"DisplayMessage":"Task received by client","Details":{},"FailsTask":false,"RestartReason":"","SetupError":"","DriverError":"","DriverMessage":"","ExitCode":0,"Signal":0,"Message":"","KillReason":"","KillTimeout":0,"KillError":"","StartDelay":0,"DownloadError":"","ValidationError":"","DiskLimit":0,"DiskSize":0,"FailedSibling":"","VaultError":"","TaskSignalReason":"","TaskSignal":"","GenericSource":""}}
April 26th 2019, 19:57:13.746 allocation {"Name":"migration-test.dummy[0]","NodeID":"95c779e9-2039-0b22-7965-45ddaff952f6","AllocationID":"d1e64eab-3fdf-e630-2d4f-60365e2eb67a","DesiredStatus":"run","DesiredDescription":"","ClientStatus":"pending","ClientDescription":"","JobID":"migration-test","GroupName":"dummy","TaskName":"dummy","EvalID":"c9c1892a-a16f-cea2-955a-c85052ed0683","TaskState":"pending","TaskFailed":false,"TaskStartedAt":"0001-01-01T00:00:00Z","TaskFinishedAt":"0001-01-01T00:00:00Z","TaskEvent":{"Type":"Driver","Time":1556308629429057838,"DisplayMessage":"Downloading image alpine:latest","Details":{"driver_message":"Downloading image alpine:latest"},"FailsTask":false,"RestartReason":"","SetupError":"","DriverError":"","DriverMessage":"Downloading image alpine:latest","ExitCode":0,"Signal":0,"Message":"","KillReason":"","KillTimeout":0,"KillError":"","StartDelay":0,"DownloadError":"","ValidationError":"","DiskLimit":0,"DiskSize":0,"FailedSibling":"","VaultError":"","TaskSignalReason":"","TaskSignal":"","GenericSource":""}}
April 26th 2019, 19:57:13.746 allocation {"Name":"migration-test.dummy[0]","NodeID":"95c779e9-2039-0b22-7965-45ddaff952f6","AllocationID":"d1e64eab-3fdf-e630-2d4f-60365e2eb67a","DesiredStatus":"run","DesiredDescription":"","ClientStatus":"running","ClientDescription":"","JobID":"migration-test","GroupName":"dummy","TaskName":"dummy","EvalID":"c9c1892a-a16f-cea2-955a-c85052ed0683","TaskState":"running","TaskFailed":false,"TaskStartedAt":"2019-04-26T19:57:10.002139319Z","TaskFinishedAt":"0001-01-01T00:00:00Z","TaskEvent":{"Type":"Started","Time":1556308629990688080,"DisplayMessage":"Task started by client","Details":{},"FailsTask":false,"RestartReason":"","SetupError":"","DriverError":"","DriverMessage":"","ExitCode":0,"Signal":0,"Message":"","KillReason":"","KillTimeout":0,"KillError":"","StartDelay":0,"DownloadError":"","ValidationError":"","DiskLimit":0,"DiskSize":0,"FailedSibling":"","VaultError":"","TaskSignalReason":"","TaskSignal":"","GenericSource":""}}
April 26th 2019, 19:57:16.731 deployment {"ID":"063a0e51-51e4-6a92-160b-0cb83f2cddef","Namespace":"default","JobID":"migration-test","JobVersion":1,"JobModifyIndex":5240337,"JobCreateIndex":5240326,"TaskGroups":{"dummy":{"PlacedCanaries":["d1e64eab-3fdf-e630-2d4f-60365e2eb67a"],"AutoRevert":false,"Promoted":false,"DesiredCanaries":1,"DesiredTotal":1,"PlacedAllocs":1,"HealthyAllocs":0,"UnhealthyAllocs":0}},"Status":"running","StatusDescription":"Deployment is running but requires promotion","CreateIndex":5240339,"ModifyIndex":5240339}
April 26th 2019, 19:57:23.731 deployment {"ID":"063a0e51-51e4-6a92-160b-0cb83f2cddef","Namespace":"default","JobID":"migration-test","JobVersion":1,"JobModifyIndex":5240337,"JobCreateIndex":5240326,"TaskGroups":{"dummy":{"PlacedCanaries":["d1e64eab-3fdf-e630-2d4f-60365e2eb67a"],"AutoRevert":false,"Promoted":false,"DesiredCanaries":1,"DesiredTotal":1,"PlacedAllocs":1,"HealthyAllocs":1,"UnhealthyAllocs":0}},"Status":"running","StatusDescription":"Deployment is running but requires promotion","CreateIndex":5240339,"ModifyIndex":5240344}
April 26th 2019, 19:57:27.693 evaluation {"ID":"c9c1892a-a16f-cea2-955a-c85052ed0683","Priority":50,"Type":"service","TriggeredBy":"job-register","Namespace":"default","JobID":"migration-test","JobModifyIndex":5240337,"NodeID":"","NodeModifyIndex":0,"DeploymentID":"063a0e51-51e4-6a92-160b-0cb83f2cddef","Status":"complete","StatusDescription":"","Wait":0,"NextEval":"","PreviousEval":"","BlockedEval":"","FailedTGAllocs":null,"ClassEligibility":null,"EscapedComputedClass":false,"QuotaLimitReached":"","AnnotatePlan":false,"QueuedAllocations":{"dummy":0},"SnapshotIndex":5240338,"CreateIndex":5240338,"ModifyIndex":5240340}
April 26th 2019, 19:57:28.697 evaluation {"ID":"2931ce75-ff84-0ba1-8cbc-a71dca9c94a0","Priority":50,"Type":"service","TriggeredBy":"deployment-watcher","Namespace":"default","JobID":"migration-test","JobModifyIndex":0,"NodeID":"","NodeModifyIndex":0,"DeploymentID":"063a0e51-51e4-6a92-160b-0cb83f2cddef","Status":"complete","StatusDescription":"","Wait":0,"NextEval":"","PreviousEval":"","BlockedEval":"","FailedTGAllocs":null,"ClassEligibility":null,"EscapedComputedClass":false,"QuotaLimitReached":"","AnnotatePlan":false,"QueuedAllocations":{"dummy":0},"SnapshotIndex":5240345,"CreateIndex":5240345,"ModifyIndex":5240346}
April 26th 2019, 19:57:30.732 deployment {"ID":"063a0e51-51e4-6a92-160b-0cb83f2cddef","Namespace":"default","JobID":"migration-test","JobVersion":1,"JobModifyIndex":5240337,"JobCreateIndex":5240326,"TaskGroups":{"dummy":{"PlacedCanaries":["d1e64eab-3fdf-e630-2d4f-60365e2eb67a"],"AutoRevert":false,"Promoted":false,"DesiredCanaries":1,"DesiredTotal":1,"PlacedAllocs":1,"HealthyAllocs":1,"UnhealthyAllocs":0}},"Status":"failed","StatusDescription":"Deployment marked as failed","CreateIndex":5240339,"ModifyIndex":5240348}
April 26th 2019, 19:57:38.747 allocation {"Name":"migration-test.dummy[0]","NodeID":"95c779e9-2039-0b22-7965-45ddaff952f6","AllocationID":"d1e64eab-3fdf-e630-2d4f-60365e2eb67a","DesiredStatus":"stop","DesiredDescription":"alloc not needed due to job update","ClientStatus":"running","ClientDescription":"","JobID":"migration-test","GroupName":"dummy","TaskName":"dummy","EvalID":"c9c1892a-a16f-cea2-955a-c85052ed0683","TaskState":"running","TaskFailed":false,"TaskStartedAt":"2019-04-26T19:57:10.002139319Z","TaskFinishedAt":"0001-01-01T00:00:00Z","TaskEvent":{"Type":"Killing","Time":1556308648465885393,"DisplayMessage":"Sent interrupt. Waiting 5s before force killing","Details":{"kill_timeout":"5s"},"FailsTask":false,"RestartReason":"","SetupError":"","DriverError":"","DriverMessage":"","ExitCode":0,"Signal":0,"Message":"","KillReason":"","KillTimeout":5000000000,"KillError":"","StartDelay":0,"DownloadError":"","ValidationError":"","DiskLimit":0,"DiskSize":0,"FailedSibling":"","VaultError":"","TaskSignalReason":"","TaskSignal":"","GenericSource":""}}
April 26th 2019, 19:57:38.747 allocation {"Name":"migration-test.dummy[0]","NodeID":"95c779e9-2039-0b22-7965-45ddaff952f6","AllocationID":"d1e64eab-3fdf-e630-2d4f-60365e2eb67a","DesiredStatus":"stop","DesiredDescription":"alloc not needed due to job update","ClientStatus":"complete","ClientDescription":"","JobID":"migration-test","GroupName":"dummy","TaskName":"dummy","EvalID":"c9c1892a-a16f-cea2-955a-c85052ed0683","TaskState":"dead","TaskFailed":false,"TaskStartedAt":"2019-04-26T19:57:10.002139319Z","TaskFinishedAt":"2019-04-26T19:57:37.887955312Z","TaskEvent":{"Type":"Killed","Time":1556308653875224503,"DisplayMessage":"Task successfully killed","Details":{},"FailsTask":false,"RestartReason":"","SetupError":"","DriverError":"","DriverMessage":"","ExitCode":0,"Signal":0,"Message":"","KillReason":"","KillTimeout":0,"KillError":"","StartDelay":0,"DownloadError":"","ValidationError":"","DiskLimit":0,"DiskSize":0,"FailedSibling":"","VaultError":"","TaskSignalReason":"","TaskSignal":"","GenericSource":""}}
April 26th 2019, 19:59:43.701 evaluation {"ID":"4538e0a4-5688-4f90-570b-e898bc6ca0fd","Priority":50,"Type":"service","TriggeredBy":"deployment-watcher","Namespace":"default","JobID":"migration-test","JobModifyIndex":0,"NodeID":"","NodeModifyIndex":0,"DeploymentID":"063a0e51-51e4-6a92-160b-0cb83f2cddef","Status":"complete","StatusDescription":"","Wait":0,"NextEval":"","PreviousEval":"","BlockedEval":"","FailedTGAllocs":null,"ClassEligibility":null,"EscapedComputedClass":false,"QuotaLimitReached":"","AnnotatePlan":false,"QueuedAllocations":{"dummy":0},"SnapshotIndex":5240348,"CreateIndex":5240348,"ModifyIndex":5240350}
April 26th 2019, 19:59:43.750 allocation {"Name":"migration-test.dummy[0]","NodeID":"6c16340a-24fe-544d-a8b4-4f91fe200a8c","AllocationID":"a92f8ba6-c9cd-5789-9cb6-faa00a785eb4","DesiredStatus":"stop","DesiredDescription":"alloc is being migrated","ClientStatus":"running","ClientDescription":"","JobID":"migration-test","GroupName":"dummy","TaskName":"dummy","EvalID":"b2154fa3-3d30-9b2d-bd6e-5bd99f9f8c2d","TaskState":"running","TaskFailed":false,"TaskStartedAt":"2019-04-26T19:56:20.302155035Z","TaskFinishedAt":"0001-01-01T00:00:00Z","TaskEvent":{"Type":"Killing","Time":1556308776756957070,"DisplayMessage":"Sent interrupt. Waiting 5s before force killing","Details":{"kill_timeout":"5s"},"FailsTask":false,"RestartReason":"","SetupError":"","DriverError":"","DriverMessage":"","ExitCode":0,"Signal":0,"Message":"","KillReason":"","KillTimeout":5000000000,"KillError":"","StartDelay":0,"DownloadError":"","ValidationError":"","DiskLimit":0,"DiskSize":0,"FailedSibling":"","VaultError":"","TaskSignalReason":"","TaskSignal":"","GenericSource":""}}
April 26th 2019, 19:59:46.750 allocation {"Name":"migration-test.dummy[0]","NodeID":"6c16340a-24fe-544d-a8b4-4f91fe200a8c","AllocationID":"a92f8ba6-c9cd-5789-9cb6-faa00a785eb4","DesiredStatus":"stop","DesiredDescription":"alloc is being migrated","ClientStatus":"complete","ClientDescription":"","JobID":"migration-test","GroupName":"dummy","TaskName":"dummy","EvalID":"b2154fa3-3d30-9b2d-bd6e-5bd99f9f8c2d","TaskState":"dead","TaskFailed":false,"TaskStartedAt":"2019-04-26T19:56:20.302155035Z","TaskFinishedAt":"2019-04-26T19:59:46.124247821Z","TaskEvent":{"Type":"Killed","Time":1556308782104828752,"DisplayMessage":"Task successfully killed","Details":{},"FailsTask":false,"RestartReason":"","SetupError":"","DriverError":"","DriverMessage":"","ExitCode":0,"Signal":0,"Message":"","KillReason":"","KillTimeout":0,"KillError":"","StartDelay":0,"DownloadError":"","ValidationError":"","DiskLimit":0,"DiskSize":0,"FailedSibling":"","VaultError":"","TaskSignalReason":"","TaskSignal":"","GenericSource":""}}
April 26th 2019, 19:59:46.751 allocation {"Name":"migration-test.dummy[0]","NodeID":"95c779e9-2039-0b22-7965-45ddaff952f6","AllocationID":"f84d1301-e212-dbe6-6d48-145d4f2598ba","DesiredStatus":"run","DesiredDescription":"","ClientStatus":"pending","ClientDescription":"","JobID":"migration-test","GroupName":"dummy","TaskName":"dummy","EvalID":"d95c80b6-4e9c-572b-98a4-175941881f42","TaskState":"pending","TaskFailed":false,"TaskStartedAt":"0001-01-01T00:00:00Z","TaskFinishedAt":"0001-01-01T00:00:00Z","TaskEvent":{"Type":"Driver","Time":1556308786222784816,"DisplayMessage":"Downloading image alpine:latest","Details":{"driver_message":"Downloading image alpine:latest"},"FailsTask":false,"RestartReason":"","SetupError":"","DriverError":"","DriverMessage":"Downloading image alpine:latest","ExitCode":0,"Signal":0,"Message":"","KillReason":"","KillTimeout":0,"KillError":"","StartDelay":0,"DownloadError":"","ValidationError":"","DiskLimit":0,"DiskSize":0,"FailedSibling":"","VaultError":"","TaskSignalReason":"","TaskSignal":"","GenericSource":""}}
April 26th 2019, 19:59:46.751 allocation {"Name":"migration-test.dummy[0]","NodeID":"95c779e9-2039-0b22-7965-45ddaff952f6","AllocationID":"f84d1301-e212-dbe6-6d48-145d4f2598ba","DesiredStatus":"run","DesiredDescription":"","ClientStatus":"pending","ClientDescription":"","JobID":"migration-test","GroupName":"dummy","TaskName":"dummy","EvalID":"d95c80b6-4e9c-572b-98a4-175941881f42","TaskState":"pending","TaskFailed":false,"TaskStartedAt":"0001-01-01T00:00:00Z","TaskFinishedAt":"0001-01-01T00:00:00Z","TaskEvent":{"Type":"Task Setup","Time":1556308786210286918,"DisplayMessage":"Building Task Directory","Details":{"message":"Building Task Directory"},"FailsTask":false,"RestartReason":"","SetupError":"","DriverError":"","DriverMessage":"","ExitCode":0,"Signal":0,"Message":"Building Task Directory","KillReason":"","KillTimeout":0,"KillError":"","StartDelay":0,"DownloadError":"","ValidationError":"","DiskLimit":0,"DiskSize":0,"FailedSibling":"","VaultError":"","TaskSignalReason":"","TaskSignal":"","GenericSource":""}}
April 26th 2019, 19:59:46.751 allocation {"Name":"migration-test.dummy[0]","NodeID":"95c779e9-2039-0b22-7965-45ddaff952f6","AllocationID":"f84d1301-e212-dbe6-6d48-145d4f2598ba","DesiredStatus":"run","DesiredDescription":"","ClientStatus":"pending","ClientDescription":"","JobID":"migration-test","GroupName":"dummy","TaskName":"dummy","EvalID":"d95c80b6-4e9c-572b-98a4-175941881f42","TaskState":"pending","TaskFailed":false,"TaskStartedAt":"0001-01-01T00:00:00Z","TaskFinishedAt":"0001-01-01T00:00:00Z","TaskEvent":{"Type":"Received","Time":1556308786210229588,"DisplayMessage":"Task received by client","Details":{},"FailsTask":false,"RestartReason":"","SetupError":"","DriverError":"","DriverMessage":"","ExitCode":0,"Signal":0,"Message":"","KillReason":"","KillTimeout":0,"KillError":"","StartDelay":0,"DownloadError":"","ValidationError":"","DiskLimit":0,"DiskSize":0,"FailedSibling":"","VaultError":"","TaskSignalReason":"","TaskSignal":"","GenericSource":""}}
April 26th 2019, 19:59:47.751 allocation {"Name":"migration-test.dummy[0]","NodeID":"95c779e9-2039-0b22-7965-45ddaff952f6","AllocationID":"f84d1301-e212-dbe6-6d48-145d4f2598ba","DesiredStatus":"run","DesiredDescription":"","ClientStatus":"running","ClientDescription":"","JobID":"migration-test","GroupName":"dummy","TaskName":"dummy","EvalID":"d95c80b6-4e9c-572b-98a4-175941881f42","TaskState":"running","TaskFailed":false,"TaskStartedAt":"2019-04-26T19:59:46.952059911Z","TaskFinishedAt":"0001-01-01T00:00:00Z","TaskEvent":{"Type":"Started","Time":1556308786940473867,"DisplayMessage":"Task started by client","Details":{},"FailsTask":false,"RestartReason":"","SetupError":"","DriverError":"","DriverMessage":"","ExitCode":0,"Signal":0,"Message":"","KillReason":"","KillTimeout":0,"KillError":"","StartDelay":0,"DownloadError":"","ValidationError":"","DiskLimit":0,"DiskSize":0,"FailedSibling":"","VaultError":"","TaskSignalReason":"","TaskSignal":"","GenericSource":""}}
```

Contributor guide

No contributing guide indexed for this repository

Research direction

Reproduce the migration scenario using the provided passing and failing job definitions: deploy with a canary, fail the update, then drain the node and inspect the migrated allocation's job version. Trace the canary deployment, failed deployment state, and allocation migration paths; done means the migrated allocation uses the last successful deployment version rather than the failed one.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.