Transport: error while dialing: dial unix /tmp/plugin connect: no such file or directory
- Dominant language
- Go
- Stars
- 17k
- Forks
- 2.1k
- Avg merge
- 1d 9h
- Merged PRs (30d)
- 105
Description
### Nomad version
Nomad v0.11.0 (5f8fe0afc894d254e4d3baaeaee1c7a70a44fdc6)
### Operating system and Environment details
Centos 7 AWS
Linux 3.10.0-1062.el7.x86_64
glibc-2.17-292.el7.x86_64
### Issue
After upgrade to 0.11 we started to observe lot of errors in nomad log related to raw_exec plugin. This appears to happen for all our periodic (batch) jobs and service using raw_exec. It started to appear after upgrade to 0.11. Clearing nomad DB and setting up cluster from scratch doesn't help. Restaritng node doesn't help either.
Whats also odd is in /tmp folder there are thousands of /tmp/plugin files, even after shutting down nomad agent. All of them are socket files of size 0
This doesn't seem to have effect on the execution of the jobs or services. They run ok, just the logs are flooded with errors.
### Reproduction steps
Submit any job that is periodic or service with task that relies on driver raw_exec and executes any binary
### Job file (if appropriate)
```
job "minute" {
datacenters = ["us-west-2"]
type = "batch"
periodic {
cron = "* * * * *"
prohibit_overlap = true
}
constraint {
attribute = "${meta.role}"
set_contains = "batch"
}
group "minute" {
count = 1
ephemeral_disk {
size = "150"
}
restart {
mode = "fail"
attempts = 0
}
task "minute" {
resources {
memory = 100
cpu = 200
}
driver = "raw_exec"
config {
command = "su"
args = ["-", "appuser", "-c", "/var/www/batch/runbatch.sh minute"]
}
}
}
}
```
### Nomad Client logs (if appropriate)
```
nomad-1587465935423358023.log:2020-04-22T10:40:47.916Z [ERROR] client.driver_mgr.raw_exec: error receiving stream from Stats executor RPC, closing stream: alloc_id=9e728345-179a-e11f-0dec-52823f363cad driver=raw_exec task_name=minute error="rpc error: code = Unavailable desc = transport is closing"
nomad-1587465935423358023.log:2020-04-22T10:40:47.916Z [ERROR] client.alloc_runner.task_runner.task_hook.stats_hook: failed to start stats collection for task: alloc_id=9e728345-179a-e11f-0dec-52823f363cad task=minute error="rpc error: code = Unavailable desc = connection error: desc = "transport: error while dialing: dial unix /tmp/plugin541251051: connect: no such file or directory""
```
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.