hashicorp / hashicorp/nomad

Sometimes nomad may place task with docker driver on host where docker not avaliable

Open
#5,485 5 comments 0 reactions 0 assignees View on GitHub
stage/needs-verification theme/client type/bug
Dominant language
Go
Stars
17k
Forks
2.1k
Avg merge
1d 9h
Merged PRs (30d)
105

Description

### Nomad version
Nomad v0.8.6 (ab54ebcfcde062e9482558b7c052702d4cb8aa1b+CHANGES)

### Issue
In our environment we have hosts where docker not installed, an we observed that nomad may place tack with docker driver on nodes where docker driver not present

Here is example for node with ID `8be15688` (there is only `exec,raw_exec` drivers presnt on it)
```
ruslan@ip-172-30-0-53:~$ nomad status 8be15688
ID = 8be15688
Name = ip-172-30-2-131
Class = openresty
DC = aws
Drain = false
Eligibility = eligible
Status = ready
Uptime = 2178h36m4s
Driver Status = exec,raw_exec

Node Events
Time Subsystem Message
2019-01-30T17:22:22+03:00 Cluster Node reregistered by heartbeat
2019-01-30T17:22:22+03:00 Cluster Node heartbeat missed
2018-12-27T22:03:12+03:00 Cluster Node registered

Allocated Resources
CPU Memory Disk IOPS
800/5900 MHz 1.9 GiB/3.4 GiB 300 MiB/14 GiB 0/0

Allocation Resource Utilization
CPU Memory
1109/5900 MHz 743 MiB/3.4 GiB

Host Resource Utilization
CPU Memory Disk
1507/6000 MHz 1.3 GiB/3.6 GiB 1.6 GiB/15 GiB

Allocations
ID Node ID Task Group Version Desired Status Created Modified
307d08bd 8be15688 dynamotoelastic 22 run failed 6m46s ago 6m8s ago
0f374e3c 8be15688 nginx 2 run running 3mo18h ago 2mo27d ago
```

As you can see allocation `307d08bd` have failed state, with follow events for it

```
Recent Events:
Time Type Description
2019-03-28T16:32:01+03:00 Not Restarting Exceeded allowed attempts 2 in interval 30m0s and mode is "fail"
2019-03-28T16:32:01+03:00 Driver Failure failed to initialize task "dynamotoelastic" for alloc "307d08bd-12c4-5951-41d7-164c00e4c99e": Failed to pull `027482006778.dkr.ecr.us-east-1.amazonaws.com/playrix/dynamodbtoelastic:p17`: dial unix /var/run/docker.sock: connect: no such file or directory
2019-03-28T16:32:01+03:00 Driver Downloading image 027482006778.dkr.ecr.us-east-1.amazonaws.com/playrix/dynamodbtoelastic:p17
2019-03-28T16:31:43+03:00 Restarting Task restarting in 18.228554529s
2019-03-28T16:31:43+03:00 Driver Failure failed to initialize task "dynamotoelastic" for alloc "307d08bd-12c4-5951-41d7-164c00e4c99e": Failed to pull `027482006778.dkr.ecr.us-east-1.amazonaws.com/playrix/dynamodbtoelastic:p17`: dial unix /var/run/docker.sock: connect: no such file or directory
2019-03-28T16:31:43+03:00 Driver Downloading image 027482006778.dkr.ecr.us-east-1.amazonaws.com/playrix/dynamodbtoelastic:p17
2019-03-28T16:31:28+03:00 Restarting Task restarting in 15.223446572s
2019-03-28T16:31:28+03:00 Driver Failure failed to initialize task "dynamotoelastic" for alloc "307d08bd-12c4-5951-41d7-164c00e4c99e": Failed to pull `027482006778.dkr.ecr.us-east-1.amazonaws.com/playrix/dynamodbtoelastic:p17`: dial unix /var/run/docker.sock: connect: no such file or directory
2019-03-28T16:31:28+03:00 Driver Downloading image 027482006778.dkr.ecr.us-east-1.amazonaws.com/playrix/dynamodbtoelastic:p17
2019-03-28T16:31:28+03:00 Task Setup Building Task Directory
```

this allocation require docker driver and it is logical that it could not run on the node without Docker, finally nomad reschedule policy can place task on right node, but i think that is fully wrong that it even not must trying to place task with docker dependency on node without docker driver

Contributor guide

No contributing guide indexed for this repository

Research direction

Reproduce the reported placement with a node exposing only exec and raw_exec drivers and a task requiring Docker. Start by tracing how Nomad filters eligible nodes by task driver availability, then add a regression test for the scheduling decision. Done means Docker tasks are not assigned to nodes without the Docker driver.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker, go
Domain
distributed-systems, infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.