hashicorp / hashicorp/nomad

Artifacts are not executable

Open
#2,676 7 comments 2 reactions 0 assignees View on GitHub
theme/client type/enhancement
Dominant language
Go
Stars
17k
Forks
2.1k
Avg merge
1d 9h
Merged PRs (30d)
105

Description

### Nomad version
Nomad v0.5.6

### Operating system and Environment details
* Ubuntu 16.04.2 LTS (Xenial Xerus)
* Docker 17.05.0-ce

### Issue
Downloaded artifacts are not executable in a docker container. It works with raw_fork but not with the Docker driver.
I get the following message when using Ubuntu with Vagrant. (I also tested it with Docker for Mac):

```
f0b494360b302168d8334382dbba185c66b77143729a4c1b6b223d10e44a26cd: API error (500): {"message":"oci runtime error: container_linux.go:247: starting container process caused \"exec: \\\"./local/app\\\": permission denied\"\n"}
```

### Reproduction steps

My go program I'm trying to run as "app":
```go
package main

import "log"

func main() {
log.Println("DONE! :)")
}
```

I build this program with the following command:
```
GOOS=linux GOARCH=amd64 go build -o app app.go
```

Then I serve the binary via a simple HTTP sever and run the job file (below).

### Nomad Server logs (if appropriate)
```
2017/05/26 19:21:59.951926 [DEBUG] http: Request /v1/evaluation/2cfa116e-4c0b-3b14-6c22-84acf6a5137f/allocations (157.131µs)
2017/05/26 19:22:00.999816 [DEBUG] driver.docker: docker pull alpine:latest succeeded
2017/05/26 19:22:01.001897 [DEBUG] driver.docker: image "alpine" (sha256:a41a7446062d197dd4b21b38122dcc7b2399deb0750c4110925a7dd37c80f118) reference count incremented: 1
2017/05/26 19:22:01 [DEBUG] plugin: starting plugin: /home/ubuntu/nomad []string{"/home/ubuntu/nomad", "executor", "{\"LogFile\":\"/tmp/NomadClient799156968/1ae5668e-74c0-050d-627d-fd236faeecd1/app/executor.out\",\"LogLevel\":\"DEBUG\"}"}
2017/05/26 19:22:01 [DEBUG] plugin: waiting for RPC address for: /home/ubuntu/nomad
2017/05/26 19:22:01 [DEBUG] plugin: nomad: 2017/05/26 19:22:01 [DEBUG] plugin: plugin address: unix /tmp/plugin476509569
2017/05/26 19:22:01.043571 [DEBUG] driver.docker: Setting default logging options to syslog and unix:///tmp/plugin112291564
2017/05/26 19:22:01.043616 [DEBUG] driver.docker: Using config for logging: {Type:syslog ConfigRaw:[] Config:map[syslog-address:unix:///tmp/plugin112291564]}
2017/05/26 19:22:01.043622 [DEBUG] driver.docker: using 268435456 bytes memory for app
2017/05/26 19:22:01.043626 [DEBUG] driver.docker: using 500 cpu shares for app
2017/05/26 19:22:01.043664 [DEBUG] driver.docker: binding directories []string{"/tmp/NomadClient799156968/1ae5668e-74c0-050d-627d-fd236faeecd1/alloc:/alloc", "/tmp/NomadClient799156968/1ae5668e-74c0-050d-627d-fd236faeecd1/app/local:/local", "/tmp/NomadClient799156968/1ae5668e-74c0-050d-627d-fd236faeecd1/app/secrets:/secrets"} for app
2017/05/26 19:22:01.043676 [DEBUG] driver.docker: networking mode not specified; defaulting to bridge
2017/05/26 19:22:01.043684 [DEBUG] driver.docker: allocated port 192.168.178.29:40699 -> 40699 (mapped)
2017/05/26 19:22:01.043690 [DEBUG] driver.docker: exposed port 40699
2017/05/26 19:22:01.043744 [DEBUG] driver.docker: setting container startup command to: ./local/app
2017/05/26 19:22:01.043760 [DEBUG] driver.docker: setting container name to: app-1ae5668e-74c0-050d-627d-fd236faeecd1
2017/05/26 19:22:01.084600 [INFO] driver.docker: created container f909e9e0fc21cf502b542d24e1ce3733b97ff4dfcddd2805195f7605679e5112
2017/05/26 19:22:01.507951 [DEBUG] driver.docker: failed to start container "f909e9e0fc21cf502b542d24e1ce3733b97ff4dfcddd2805195f7605679e5112" (attempt 1): API error (500): {"message":"oci runtime error: container_linux.go:247: starting container process caused \"exec: \\\"./local/app\\\": permission denied\"\n"}
2017/05/26 19:22:02.939463 [DEBUG] driver.docker: failed to start container "f909e9e0fc21cf502b542d24e1ce3733b97ff4dfcddd2805195f7605679e5112" (attempt 2): API error (500): {"message":"oci runtime error: container_linux.go:247: starting container process caused \"exec: \\\"./local/app\\\": permission denied\"\n"}
2017/05/26 19:22:04.331313 [DEBUG] driver.docker: failed to start container "f909e9e0fc21cf502b542d24e1ce3733b97ff4dfcddd2805195f7605679e5112" (attempt 3): API error (500): {"message":"oci runtime error: container_linux.go:247: starting container process caused \"exec: \\\"./local/app\\\": permission denied\"\n"}
2017/05/26 19:22:05.746869 [DEBUG] driver.docker: failed to start container "f909e9e0fc21cf502b542d24e1ce3733b97ff4dfcddd2805195f7605679e5112" (attempt 4): API error (500): {"message":"oci runtime error: container_linux.go:247: starting container process caused \"exec: \\\"./local/app\\\": permission denied\"\n"}
2017/05/26 19:22:07.171912 [DEBUG] driver.docker: failed to start container "f909e9e0fc21cf502b542d24e1ce3733b97ff4dfcddd2805195f7605679e5112" (attempt 5): API error (500): {"message":"oci runtime error: container_linux.go:247: starting container process caused \"exec: \\\"./local/app\\\": permission denied\"\n"}
2017/05/26 19:22:08.579853 [DEBUG] driver.docker: failed to start container "f909e9e0fc21cf502b542d24e1ce3733b97ff4dfcddd2805195f7605679e5112" (attempt 6): API error (500): {"message":"oci runtime error: container_linux.go:247: starting container process caused \"exec: \\\"./local/app\\\": permission denied\"\n"}
2017/05/26 19:22:08.579976 [ERR] driver.docker: failed to start container f909e9e0fc21cf502b542d24e1ce3733b97ff4dfcddd2805195f7605679e5112: API error (500): {"message":"oci runtime error: container_linux.go:247: starting container process caused \"exec: \\\"./local/app\\\": permission denied\"\n"}
2017/05/26 19:22:08 [DEBUG] plugin: /home/ubuntu/nomad: plugin process exited
2017/05/26 19:22:08.582071 [WARN] client: failed to start task "app" for alloc "1ae5668e-74c0-050d-627d-fd236faeecd1": Failed to start container f909e9e0fc21cf502b542d24e1ce3733b97ff4dfcddd2805195f7605679e5112: API error (500): {"message":"oci runtime error: container_linux.go:247: starting container process caused \"exec: \\\"./local/app\\\": permission denied\"\n"}
2017/05/26 19:22:08.582646 [INFO] client: Not restarting task: app for alloc: 1ae5668e-74c0-050d-627d-fd236faeecd1
2017/05/26 19:22:08.582877 [DEBUG] driver.docker: image id "sha256:a41a7446062d197dd4b21b38122dcc7b2399deb0750c4110925a7dd37c80f118" reference count decremented: 0
2017/05/26 19:22:08.583423 [INFO] client: marking allocation 1ae5668e-74c0-050d-627d-fd236faeecd1 for GC
2017/05/26 19:22:08.584152 [INFO] client: marking allocation 1ae5668e-74c0-050d-627d-fd236faeecd1 for GC
2017/05/26 19:22:08.746745 [DEBUG] client: updated allocations at index 12 (total 1) (pulled 0) (filtered 1)
2017/05/26 19:22:08.746897 [DEBUG] client: allocs: (added 0) (removed 0) (updated 0) (ignore 1)
```

### Nomad Client logs (if appropriate)
```
➜ Desktop ./nomad alloc-status --address=http://192.168.178.29:4646 1ae5668e
ID = 1ae5668e
Eval ID = 2cfa116e
Name = app.app[0]
Node ID = e6e35f7b
Job ID = app
Client Status = failed
Client Description =
Desired Status = run
Desired Description =
Created At = 05/26/17 21:21:58 CEST

Task "app" is "dead"
Task Resources
CPU Memory Disk IOPS Addresses
500 MHz 256 MiB 300 MiB 0 db: 192.168.178.29:40699

Recent Events:
Time Type Description
05/26/17 21:22:08 CEST Not Restarting Policy allows no restarts
05/26/17 21:22:08 CEST Driver Failure failed to start task "app" for alloc "1ae5668e-74c0-050d-627d-fd236faeecd1": Failed to start container f909e9e0fc21cf502b542d24e1ce3733b97ff4dfcddd2805195f7605679e5112: API error (500): {"message":"oci runtime error: container_linux.go:247: starting container process caused \"exec: \\\"./local/app\\\": permission denied\"\n"}
05/26/17 21:21:58 CEST Driver Downloading image alpine:latest
05/26/17 21:21:58 CEST Downloading Artifacts Client is downloading artifacts
05/26/17 21:21:58 CEST Task Setup Building Task Directory
05/26/17 21:21:58 CEST Received Task received by client
```

### Job file (if appropriate)
```
job "app" {
datacenters = ["dc1"]

type = "batch"

update {
stagger = "10s"
max_parallel = 1
}

group "app" {
count = 1

restart {
attempts = 0
interval = "5m"
delay = "25s"
mode = "fail"
}

task "app" {
driver = "docker"

config {
image = "alpine"
command = "./local/app"
}

artifact {
source = "http://localhost:8333/app"
}

logs {
max_files = 1
max_file_size = 15
}

resources {
cpu = 500 # 500 MHz
memory = 256 # 256MB
network {
mbits = 10
port "db" {}
}
}
}
}
}
```

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reproducing the reported job with Nomad v0.5.6, the Docker driver, the Alpine image, and the HTTP artifact source. Trace artifact download and Docker task startup handling; done means the downloaded Go binary can be launched through command = "./local/app" without a permission-denied error.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker, go
Domain
devops, infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.