Sometimes nomad may place task with docker driver on host where docker not avaliable
- Dominant language
- Go
- Stars
- 17k
- Forks
- 2.1k
- Avg merge
- 1d 9h
- Merged PRs (30d)
- 105
Description
### Nomad version
Nomad v0.8.6 (ab54ebcfcde062e9482558b7c052702d4cb8aa1b+CHANGES)
### Issue
In our environment we have hosts where docker not installed, an we observed that nomad may place tack with docker driver on nodes where docker driver not present
Here is example for node with ID `8be15688` (there is only `exec,raw_exec` drivers presnt on it)
```
ruslan@ip-172-30-0-53:~$ nomad status 8be15688
ID = 8be15688
Name = ip-172-30-2-131
Class = openresty
DC = aws
Drain = false
Eligibility = eligible
Status = ready
Uptime = 2178h36m4s
Driver Status = exec,raw_exec
Node Events
Time Subsystem Message
2019-01-30T17:22:22+03:00 Cluster Node reregistered by heartbeat
2019-01-30T17:22:22+03:00 Cluster Node heartbeat missed
2018-12-27T22:03:12+03:00 Cluster Node registered
Allocated Resources
CPU Memory Disk IOPS
800/5900 MHz 1.9 GiB/3.4 GiB 300 MiB/14 GiB 0/0
Allocation Resource Utilization
CPU Memory
1109/5900 MHz 743 MiB/3.4 GiB
Host Resource Utilization
CPU Memory Disk
1507/6000 MHz 1.3 GiB/3.6 GiB 1.6 GiB/15 GiB
Allocations
ID Node ID Task Group Version Desired Status Created Modified
307d08bd 8be15688 dynamotoelastic 22 run failed 6m46s ago 6m8s ago
0f374e3c 8be15688 nginx 2 run running 3mo18h ago 2mo27d ago
```
As you can see allocation `307d08bd` have failed state, with follow events for it
```
Recent Events:
Time Type Description
2019-03-28T16:32:01+03:00 Not Restarting Exceeded allowed attempts 2 in interval 30m0s and mode is "fail"
2019-03-28T16:32:01+03:00 Driver Failure failed to initialize task "dynamotoelastic" for alloc "307d08bd-12c4-5951-41d7-164c00e4c99e": Failed to pull `027482006778.dkr.ecr.us-east-1.amazonaws.com/playrix/dynamodbtoelastic:p17`: dial unix /var/run/docker.sock: connect: no such file or directory
2019-03-28T16:32:01+03:00 Driver Downloading image 027482006778.dkr.ecr.us-east-1.amazonaws.com/playrix/dynamodbtoelastic:p17
2019-03-28T16:31:43+03:00 Restarting Task restarting in 18.228554529s
2019-03-28T16:31:43+03:00 Driver Failure failed to initialize task "dynamotoelastic" for alloc "307d08bd-12c4-5951-41d7-164c00e4c99e": Failed to pull `027482006778.dkr.ecr.us-east-1.amazonaws.com/playrix/dynamodbtoelastic:p17`: dial unix /var/run/docker.sock: connect: no such file or directory
2019-03-28T16:31:43+03:00 Driver Downloading image 027482006778.dkr.ecr.us-east-1.amazonaws.com/playrix/dynamodbtoelastic:p17
2019-03-28T16:31:28+03:00 Restarting Task restarting in 15.223446572s
2019-03-28T16:31:28+03:00 Driver Failure failed to initialize task "dynamotoelastic" for alloc "307d08bd-12c4-5951-41d7-164c00e4c99e": Failed to pull `027482006778.dkr.ecr.us-east-1.amazonaws.com/playrix/dynamodbtoelastic:p17`: dial unix /var/run/docker.sock: connect: no such file or directory
2019-03-28T16:31:28+03:00 Driver Downloading image 027482006778.dkr.ecr.us-east-1.amazonaws.com/playrix/dynamodbtoelastic:p17
2019-03-28T16:31:28+03:00 Task Setup Building Task Directory
```
this allocation require docker driver and it is logical that it could not run on the node without Docker, finally nomad reschedule policy can place task on right node, but i think that is fully wrong that it even not must trying to place task with docker dependency on node without docker driver
Contributor guide
No contributing guide indexed for this repository
Research direction
Reproduce the reported placement with a node exposing only exec and raw_exec drivers and a task requiring Docker. Start by tracing how Nomad filters eligible nodes by task driver availability, then add a regression test for the scheduling decision. Done means Docker tasks are not assigned to nodes without the Docker driver.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- docker, go
- Domain
- distributed-systems, infrastructure
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100