hashicorp / hashicorp/nomad

Transport: error while dialing: dial unix /tmp/plugin connect: no such file or directory

Open
#7,776 7 comments 0 reactions 0 assignees View on GitHub
theme/logging type/bug
Dominant language
Go
Stars
17k
Forks
2.1k
Avg merge
1d 9h
Merged PRs (30d)
105

Description

### Nomad version
Nomad v0.11.0 (5f8fe0afc894d254e4d3baaeaee1c7a70a44fdc6)

### Operating system and Environment details
Centos 7 AWS
Linux 3.10.0-1062.el7.x86_64
glibc-2.17-292.el7.x86_64

### Issue
After upgrade to 0.11 we started to observe lot of errors in nomad log related to raw_exec plugin. This appears to happen for all our periodic (batch) jobs and service using raw_exec. It started to appear after upgrade to 0.11. Clearing nomad DB and setting up cluster from scratch doesn't help. Restaritng node doesn't help either.

Whats also odd is in /tmp folder there are thousands of /tmp/plugin files, even after shutting down nomad agent. All of them are socket files of size 0

This doesn't seem to have effect on the execution of the jobs or services. They run ok, just the logs are flooded with errors.

### Reproduction steps
Submit any job that is periodic or service with task that relies on driver raw_exec and executes any binary

### Job file (if appropriate)
```
job "minute" {
datacenters = ["us-west-2"]
type = "batch"
periodic {
cron = "* * * * *"
prohibit_overlap = true
}
constraint {
attribute = "${meta.role}"
set_contains = "batch"
}
group "minute" {
count = 1
ephemeral_disk {
size = "150"
}
restart {
mode = "fail"
attempts = 0
}
task "minute" {
resources {
memory = 100
cpu = 200
}
driver = "raw_exec"
config {
command = "su"
args = ["-", "appuser", "-c", "/var/www/batch/runbatch.sh minute"]
}
}
}
}
```

### Nomad Client logs (if appropriate)
```
nomad-1587465935423358023.log:2020-04-22T10:40:47.916Z [ERROR] client.driver_mgr.raw_exec: error receiving stream from Stats executor RPC, closing stream: alloc_id=9e728345-179a-e11f-0dec-52823f363cad driver=raw_exec task_name=minute error="rpc error: code = Unavailable desc = transport is closing"
nomad-1587465935423358023.log:2020-04-22T10:40:47.916Z [ERROR] client.alloc_runner.task_runner.task_hook.stats_hook: failed to start stats collection for task: alloc_id=9e728345-179a-e11f-0dec-52823f363cad task=minute error="rpc error: code = Unavailable desc = connection error: desc = "transport: error while dialing: dial unix /tmp/plugin541251051: connect: no such file or directory""
```

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.