Nomad depends on gnu implementation of df
- Dominant language
- Go
- Stars
- 17k
- Forks
- 2.1k
- Avg merge
- 1d 9h
- Merged PRs (30d)
- 105
Description
### Nomad version
1.2.6, freshly downloaded from https://releases.hashicorp.com/nomad/1.2.6/nomad_1.2.6_linux_amd64.zip
### Operating system and Environment details
Minimal linux kernel with custom init, built & run using `make test` from https://github.com/finwo/nomados/
Start command:
- `/sbin/nomad agent -dev -log-level=TRACE -config=/etc/nomad/init.json`
- `/sbin/nomad agent -log-level=TRACE -config=/etc/nomad/init.json`
Config:
```json
{
"data_dir": "/var/lib/nomad/data",
"datacenter": "dc1",
"client": {
"enabled": true,
"options": {
"user.denylist": "",
"user.checked_drivers": "exec,raw_exec"
},
"server_join": {
"retry_join": ["nomad", "10.1.1.80"],
"retry_max": 10,
"retry_interval": "1s"
},
"state_dir": "/var/lib/nomad/client"
},
"server": {
"enabled": false
},
"log_level": "TRACE",
"bind_addr": "0.0.0.0"
}
```
### Issue
`agent: error starting agent: error="client setup failed: fingerprinting failed: failed to determine disk space for /var/lib/nomad/data/alloc: failed to determine mount point"`
Nomad throws an error early during startup as shown above, stating it can't determine the available disk space or mount point, when running in a minimal environment on ramdisk (as in the linked repository).
### Reproduction steps
Clone the linked repository, run `make test` in the root of the repsitory, wait for it all to build, start and crash
#### Expected Result
A virtual machine running as a nomad client which joins the server that's already running in my network
#### Actual Result
After getting an IP and sarting nomad, nomad throws an error (`agent: error starting agent: error="client setup failed: fingerprinting failed: failed to determine disk space for /var/lib/nomad/data/alloc: failed to determine mount point"`) stating it can't determine the mount point for it's alloc directory.
The problem arises both with and without the `-dev` flag, although without the flag it depends on `/sbin/ip` to be present before the error is thrown.
### Job file (if appropriate)
n/a
### Nomad Server logs (if appropriate)
n/a
### Nomad Client logs (if appropriate)
```
==> Loaded configuration from /etc/nomad/init.json
==> Starting Nomad agent...
==> Error starting agent: client setup failed: fingerprinting failed: failed to determine disk space for /var/lib/nomad/data/alloc: failed to determine mount point for /var/lib/nomad/data/alloc
2022-04-02T17:15:16.255Z [WARN] agent.plugin_loader: skipping external plugins since plugin_dir doesn't exist: plugin_dir=/var/lib/nomad/data/plugins
2022-04-02T17:15:16.256Z [DEBUG] agent.plugin_loader.docker: using client connection initialized from environment: plugin_dir=/var/lib/nomad/data/plugins
2022-04-02T17:15:16.256Z [DEBUG] agent.plugin_loader.docker: using client connection initialized from environment: plugin_dir=/var/lib/nomad/data/plugins
2022-04-02T17:15:16.256Z [INFO] agent: detected plugin: name=java type=driver plugin_version=0.1.0
2022-04-02T17:15:16.256Z [INFO] agent: detected plugin: name=docker type=driver plugin_version=0.1.0
2022-04-02T17:15:16.256Z [INFO] agent: detected plugin: name=raw_exec type=driver plugin_version=0.1.0
2022-04-02T17:15:16.256Z [INFO] agent: detected plugin: name=exec type=driver plugin_version=0.1.0
2022-04-02T17:15:16.256Z [INFO] agent: detected plugin: name=qemu type=driver plugin_version=0.1.0
2022-04-02T17:15:16.256Z [INFO] client: using state directory: state_dir=/var/lib/nomad/client
2022-04-02T17:15:16.256Z [INFO] client: using alloc directory: alloc_dir=/var/lib/nomad/data/alloc
2022-04-02T17:15:16.256Z [INFO] client: using dynamic ports: min=20000 max=32000 reserved=""
2022-04-02T17:15:16.256Z [WARN] client: could not initialize cpuset cgroup subsystem, cpuset management disabled: error="not implemented for cgroup v2 unified hierarchy"
2022-04-02T17:15:16.256Z [DEBUG] client.fingerprint_mgr: built-in fingerprints: fingerprinters=["arch", "bridge", "cgroup", "cni", "consul", "cpu", "host", "memory", "network", "nomad", "signal", "storage",]
2022-04-02T17:15:16.257Z [WARN] client.fingerprint_mgr: failed to detect bridge kernel module, bridge network mode disabled:
error=
| 3 errors occurred:
| * failed to open /proc/modules: open /proc/modules: no such file or directory
| * failed to open /lib/modules/5.17.0+/modules.builtin: open /lib/modules/5.17.0+/modules.builtin: no such file or directory
| * failed to open /lib/modules/5.17.0+/modules.dep: open /lib/modules/5.17.0+/modules.dep: no such file or directory
|
2022-04-02T17:15:16.257Z [INFO] client.fingerprint_mgr.cgroup: cgroups are available
2022-04-02T17:15:16.257Z [DEBUG] client.fingerprint_mgr: CNI config dir is not set or does not exist, skipping: cni_config_dir=/opt/cni/config
2022-04-02T17:15:16.257Z [TRACE] agent.plugin_loader.exec: task event loop shutdown: plugin_dir=/var/lib/nomad/data/plugins
2022-04-02T17:15:16.257Z [TRACE] agent.plugin_loader.qemu: task event loop shutdown: plugin_dir=/var/lib/nomad/data/plugins
2022-04-02T17:15:16.257Z [TRACE] agent.plugin_loader.java: task event loop shutdown: plugin_dir=/var/lib/nomad/data/plugins
2022-04-02T17:15:16.257Z [TRACE] agent.plugin_loader.docker: task event loop shutdown: plugin_dir=/var/lib/nomad/data/plugins
2022-04-02T17:15:16.257Z [TRACE] agent.plugin_loader.raw_exec: task event loop shutdown: plugin_dir=/var/lib/nomad/data/plugins
2022-04-02T17:15:16.257Z [TRACE] agent.plugin_loader.java: task event loop shutdown: plugin_dir=/var/lib/nomad/data/plugins
2022-04-02T17:15:16.257Z [TRACE] agent.plugin_loader.docker: task event loop shutdown: plugin_dir=/var/lib/nomad/data/plugins
2022-04-02T17:15:16.257Z [TRACE] agent.plugin_loader.raw_exec: task event loop shutdown: plugin_dir=/var/lib/nomad/data/plugins
2022-04-02T17:15:16.257Z [TRACE] agent.plugin_loader.exec: task event loop shutdown: plugin_dir=/var/lib/nomad/data/plugins
2022-04-02T17:15:16.257Z [TRACE] agent.plugin_loader.qemu: task event loop shutdown: plugin_dir=/var/lib/nomad/data/plugins
2022-04-02T17:15:16.257Z [DEBUG] client.fingerprint_mgr: fingerprinting periodically: fingerprinter=cgroup period=15s
2022-04-02T17:15:16.257Z [DEBUG] client.fingerprint_mgr.cpu: detected cpu frequency: MHz=1608
2022-04-02T17:15:16.257Z [DEBUG] client.fingerprint_mgr.cpu: detected core count: cores=1
2022-04-02T17:15:16.257Z [WARN] client.fingerprint_mgr.cpu: failed to detect set of reservable cores: error="not implemented for cgroup v2 unified hierarchy"
2022-04-02T17:15:16.258Z [DEBUG] client.fingerprint_mgr: fingerprinting periodically: fingerprinter=consul period=15s
2022-04-02T17:15:16.258Z [DEBUG] client.fingerprint_mgr.network: link speed detected: interface=eth0 mbits=1000
2022-04-02T17:15:16.258Z [DEBUG] client.fingerprint_mgr.network: detected interface IP: interface=eth0 IP=10.0.2.15
2022-04-02T17:15:16.258Z [DEBUG] client.fingerprint_mgr.network: detected interface IP: interface=eth0 IP=fec0::5054:ff:fe12:3456
2022-04-02T17:15:16.258Z [DEBUG] client.fingerprint_mgr.network: unable to read link speed: path=/sys/class/net/lo/speed device=lo
2022-04-02T17:15:16.258Z [DEBUG] client.fingerprint_mgr.network: link speed could not be detected, falling back to default speed: interface=lo mbits=1000
2022-04-02T17:15:16.261Z [DEBUG] client.fingerprint_mgr.network: unable to read link speed: path=/sys/class/net/sit0/speed device=sit0
2022-04-02T17:15:16.261Z [DEBUG] client.fingerprint_mgr.network: link speed could not be detected, falling back to default speed: interface=sit0 mbits=1000
2022-04-02T17:15:16.261Z [ERROR] agent: error starting agent: error="client setup failed: fingerprinting failed: failed to determine disk space for /var/lib/nomad/data/alloc: failed to determine mount point"
```
Contributor guide
No contributing guide indexed for this repository
Research direction
Use the supplied Nomad 1.2.6 command and configuration with finwo/nomados and run `make test` to reproduce the mount-point failure. Trace client startup from the reported storage fingerprinting error and compare behavior in the minimal Linux environment with a normal Linux host. Done means the client starts successfully in the reported environment without regressing disk-space detection.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, linux
- Domain
- infrastructure, operating-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100