hashicorp / hashicorp/nomad

Environment fingerprinters allow bogus values

Open
#9,698 2 comments 3 reactions 0 assignees View on GitHub
stage/needs-discussion theme/fingerprint type/bug
Dominant language
Go
Stars
17k
Forks
2.1k
Avg merge
1d 9h
Merged PRs (30d)
105

Description

### Nomad version
1.0.1

### Operating system and Environment details
Ubuntu 18.04 baremetal (non cloud)

### Issue

When the environment fingerprinters try to obtain the network details, it is possible to end up with some bogus values as described in:

https://discuss.hashicorp.com/t/network-fingerprinting-calling-my-home-router/19023

In this specific case the values were populated with a html page from the home router probably because it was retuning a 200 status code on non existing pages and confusing the client thinking it had a successful request and setting the body of the request as the value.

### Reproduction steps

* Run Nomad in a non cloud environment with the env_fingerprinters on (default).
* A router address that return 200 on any request with some unwanted body
* Logs and UI will show the bogus body printed out.

### Nomad Client logs (if appropriate)
```
Dec 18 09:03:45 nomadnode01 systemd[1]: Started Nomad Client.
Dec 18 09:03:45 nomadnode01 nomad[6979]: ==> WARNING: Bootstrap mode enabled! Potentially unsafe operation.
Dec 18 09:03:45 nomadnode01 nomad[6979]: ==> Loaded configuration from /etc/nomad.d/client.hcl, /etc/nomad.d/server.hcl
Dec 18 09:03:45 nomadnode01 nomad[6979]: ==> Starting Nomad agent...
Dec 18 09:03:46 nomadnode01 nomad[6979]: ==> Nomad agent configuration:
Dec 18 09:03:46 nomadnode01 nomad[6979]: Advertise Addrs: HTTP: 192.168.1.11:4646; RPC: 192.168.1.11:4647; Serf: 192.168.1.11:4648
Dec 18 09:03:46 nomadnode01 nomad[6979]: Bind Addrs: HTTP: 192.168.1.11:4646; RPC: 192.168.1.11:4647; Serf: 192.168.1.11:4648
Dec 18 09:03:46 nomadnode01 nomad[6979]: Client: true
Dec 18 09:03:46 nomadnode01 nomad[6979]: Log Level: DEBUG
Dec 18 09:03:46 nomadnode01 nomad[6979]: Region: global (DC: dc01)
Dec 18 09:03:46 nomadnode01 nomad[6979]: Server: true
Dec 18 09:03:46 nomadnode01 nomad[6979]: Version: 1.0.1
Dec 18 09:03:46 nomadnode01 nomad[6979]: ==> Nomad agent started! Log data will stream in below:
Dec 18 09:03:46 nomadnode01 nomad[6979]: 2020-12-18T09:03:45.775Z [WARN] agent.plugin_loader: skipping external plugins since plugin_dir doesn't exist: plugin_dir=/var/lib/nomad/storage/server/plugins
Dec 18 09:03:46 nomadnode01 nomad[6979]: 2020-12-18T09:03:45.776Z [DEBUG] agent.plugin_loader.docker: using client connection initialized from environment: plugin_dir=/var/lib/nomad/storage/server/plugins
Dec 18 09:03:46 nomadnode01 nomad[6979]: 2020-12-18T09:03:45.776Z [INFO] agent: detected plugin: name=qemu type=driver plugin_version=0.1.0
Dec 18 09:03:46 nomadnode01 nomad[6979]: 2020-12-18T09:03:45.776Z [INFO] agent: detected plugin: name=java type=driver plugin_version=0.1.0
Dec 18 09:03:46 nomadnode01 nomad[6979]: 2020-12-18T09:03:45.776Z [INFO] agent: detected plugin: name=docker type=driver plugin_version=0.1.0
Dec 18 09:03:46 nomadnode01 nomad[6979]: 2020-12-18T09:03:45.776Z [INFO] agent: detected plugin: name=raw_exec type=driver plugin_version=0.1.0
Dec 18 09:03:46 nomadnode01 nomad[6979]: 2020-12-18T09:03:45.776Z [INFO] agent: detected plugin: name=exec type=driver plugin_version=0.1.0
Dec 18 09:03:46 nomadnode01 nomad[6979]: 2020-12-18T09:03:45.776Z [INFO] agent: detected plugin: name=nvidia-gpu type=device plugin_version=0.1.0
Dec 18 09:03:46 nomadnode01 nomad[6979]: 2020-12-18T09:03:45.794Z [INFO] nomad.raft: restored from snapshot: id=7-8351-1608257027253
Dec 18 09:03:46 nomadnode01 nomad[6979]: 2020-12-18T09:03:45.811Z [INFO] nomad.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:192.168.1.11:4647 Address:192.168.1.11:4647}]"
Dec 18 09:03:46 nomadnode01 nomad[6979]: 2020-12-18T09:03:46.469Z [INFO] client.gc: marking allocation for GC: alloc_id=5042c56f-fa29-467d-e6ad-7be5e42e4a07
Dec 18 09:03:46 nomadnode01 nomad[6979]: 2020-12-18T09:03:45.811Z [INFO] nomad.raft: entering follower state: follower="Node at 192.168.1.11:4647 [Follower]" leader=
Dec 18 09:03:46 nomadnode01 nomad[6979]: 2020-12-18T09:03:45.811Z [INFO] nomad: serf: EventMemberJoin: client.global 192.168.1.11
Dec 18 09:03:46 nomadnode01 nomad[6979]: 2020-12-18T09:03:45.811Z [INFO] nomad: starting scheduling worker(s): num_workers=4 schedulers=[service, batch, system, _core]
Dec 18 09:03:46 nomadnode01 nomad[6979]: 2020-12-18T09:03:45.811Z [WARN] nomad: serf: Failed to re-join any previously known node
Dec 18 09:03:46 nomadnode01 nomad[6979]: 2020-12-18T09:03:45.811Z [INFO] client: using state directory: state_dir=/var/lib/nomad/storage/server/client
Dec 18 09:03:46 nomadnode01 nomad[6979]: 2020-12-18T09:03:45.812Z [INFO] nomad: adding server: server="client.global (Addr: 192.168.1.11:4647) (DC: dc01)"
Dec 18 09:03:46 nomadnode01 nomad[6979]: 2020-12-18T09:03:45.812Z [INFO] client: using alloc directory: alloc_dir=/var/lib/nomad/storage/server/alloc
Dec 18 09:03:46 nomadnode01 nomad[6979]: 2020-12-18T09:03:45.813Z [DEBUG] client.fingerprint_mgr: built-in fingerprints: fingerprinters=[arch, bridge, cgroup, cni, consul, cpu, host, memory, network, nomad, signal, storage, vault, env_azure, env_aws, env_gce]
Dec 18 09:03:46 nomadnode01 nomad[6979]: 2020-12-18T09:03:45.813Z [INFO] client.fingerprint_mgr.cgroup: cgroups are available
Dec 18 09:03:46 nomadnode01 nomad[6979]: 2020-12-18T09:03:45.813Z [DEBUG] client.fingerprint_mgr: CNI config dir is not set or does not exist, skipping: cni_config_dir=
Dec 18 09:03:46 nomadnode01 nomad[6979]: 2020-12-18T09:03:45.813Z [DEBUG] client.fingerprint_mgr: fingerprinting periodically: fingerprinter=cgroup period=15s
Dec 18 09:03:46 nomadnode01 nomad[6979]: 2020-12-18T09:03:45.815Z [INFO] client.fingerprint_mgr.consul: consul agent is available
Dec 18 09:03:46 nomadnode01 nomad[6979]: 2020-12-18T09:03:45.815Z [DEBUG] client.fingerprint_mgr: fingerprinting periodically: fingerprinter=consul period=15s
Dec 18 09:03:46 nomadnode01 nomad[6979]: 2020-12-18T09:03:45.815Z [DEBUG] client.fingerprint_mgr.cpu: detected cpu frequency: MHz=3900
Dec 18 09:03:46 nomadnode01 nomad[6979]: 2020-12-18T09:03:45.815Z [DEBUG] client.fingerprint_mgr.cpu: detected core count: cores=4
Dec 18 09:03:46 nomadnode01 nomad[6979]: 2020-12-18T09:03:45.817Z [DEBUG] client.fingerprint_mgr.network: link speed detected: interface=enp0s31f6 mbits=100
Dec 18 09:03:46 nomadnode01 nomad[6979]: 2020-12-18T09:03:45.817Z [DEBUG] client.fingerprint_mgr.network: detected interface IP: interface=enp0s31f6 IP=192.168.1.11
Dec 18 09:03:46 nomadnode01 nomad[6979]: 2020-12-18T09:03:45.817Z [DEBUG] client.fingerprint_mgr.network: detected interface IP: interface=enp0s31f6 IP=2806:102e:8:2e65::9
Dec 18 09:03:46 nomadnode01 nomad[6979]: 2020-12-18T09:03:45.817Z [DEBUG] client.fingerprint_mgr.network: detected interface IP: interface=enp0s31f6 IP=2806:102e:8:2e65:e2d5:5eff:fe00:8c97
Dec 18 09:03:46 nomadnode01 nomad[6979]: 2020-12-18T09:03:45.817Z [DEBUG] client.fingerprint_mgr.network: detected interface IP: interface=enp0s31f6 IP=fdac:6175:1a7e:dd00:e2d5:5eff:fe00:8c97
Dec 18 09:03:46 nomadnode01 nomad[6979]: 2020-12-18T09:03:45.818Z [WARN] client.fingerprint_mgr.network: unable to parse speed: path=/sbin/ethtool device=lo
Dec 18 09:03:46 nomadnode01 nomad[6979]: 2020-12-18T09:03:45.818Z [DEBUG] client.fingerprint_mgr.network: unable to read link speed: path=/sys/class/net/lo/speed
Dec 18 09:03:46 nomadnode01 nomad[6979]: 2020-12-18T09:03:45.818Z [DEBUG] client.fingerprint_mgr.network: link speed could not be detected, falling back to default speed: mbits=1000
Dec 18 09:03:46 nomadnode01 nomad[6979]: 2020-12-18T09:03:45.819Z [WARN] client.fingerprint_mgr.network: unable to parse speed: path=/sbin/ethtool device=docker0
Dec 18 09:03:46 nomadnode01 nomad[6979]: 2020-12-18T09:03:45.819Z [DEBUG] client.fingerprint_mgr.network: unable to read link speed: path=/sys/class/net/docker0/speed
Dec 18 09:03:46 nomadnode01 nomad[6979]: 2020-12-18T09:03:45.819Z [DEBUG] client.fingerprint_mgr.network: link speed could not be detected, falling back to default speed: mbits=1000
Dec 18 09:03:46 nomadnode01 nomad[6979]: 2020-12-18T09:03:45.823Z [DEBUG] client.fingerprint_mgr: fingerprinting periodically: fingerprinter=vault period=15s
Dec 18 09:03:46 nomadnode01 nomad[6979]: 2020-12-18T09:03:45.858Z [DEBUG] consul.sync: sync complete: registered_services=3 deregistered_services=0 registered_checks=3 deregistered_checks=0
Dec 18 09:03:46 nomadnode01 nomad[6979]: 2020-12-18T09:03:46.025Z [DEBUG] client.fingerprint_mgr.env_aws: read an empty value: attribute="network/interfaces/macs/
Dec 18 09:03:46 nomadnode01 nomad[6979]:
Dec 18 09:03:46 nomadnode01 nomad[6979]:
Dec 18 09:03:46 nomadnode01 nomad[6979]:
Dec 18 09:03:46 nomadnode01 nomad[6979]:
Dec 18 09:03:46 nomadnode01 nomad[6979]: Waiting...
Dec 18 09:03:46 nomadnode01 nomad[6979]:
Dec 18 09:03:46 nomadnode01 nomad[6979]: var pageName = '/';
Dec 18 09:03:46 nomadnode01 nomad[6979]: top.location.replace(pageName);
Dec 18 09:03:46 nomadnode01 nomad[6979]:
Dec 18 09:03:46 nomadnode01 nomad[6979]:
Dec 18 09:03:46 nomadnode01 nomad[6979]:
Dec 18 09:03:46 nomadnode01 nomad[6979]: /ipv6s"
```

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by locating the environment fingerprinter entry points, especially the env_aws logs shown in the report, and trace how network metadata responses are accepted. Reproduce the case with a non-cloud endpoint that returns HTML with HTTP 200. Done means bogus response bodies are not recorded or exposed as fingerprint values, with regression coverage added where the relevant tests are found.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.