hashicorp / hashicorp/nomad-driver-virt

net: Nomad service port forwarding from client host machine local to VM

Open
#48 0 comments 0 reactions 0 assignees View on GitHub
stage/accepted theme/network theme/os/linux type/bug
Dominant language
Go
Stars
32
Forks
9
Avg merge
1d 10h
Merged PRs (30d)
11

Description

The current iptables rules used, as per [libvirts forwarding incoming connections doc](https://wiki.libvirt.org/Networking.html#forwarding-incoming-connections), do not allow for connections coming from the local Nomad client host to VM's running on it via the client interface IP address.

```console
root@libvirt-0:/nomad-test# nomad service info jrasell-workstation-ssh
Job ID Address Tags Node ID Alloc ID
jrasell-vm 10.0.1.17:22982 [] a400db6e 6d0386a6
root@libvirt-0:/nomad-test# ssh 10.0.1.17 -p 22982
ssh: connect to host 10.0.1.17 port 22982: Connection refused
```

The virsh console command and using the libvirt DHCP assigned address work as expected and can be used as a workaround until this issue is fixed.

```console
root@libvirt-0:/nomad-test# virsh console vm-795c791f
Connected to domain 'vm-795c791f'
Escape character is ^] (Ctrl + ])
```

```console
root@libvirt-0:/nomad-test# virsh net-dhcp-leases default
Expiry Time MAC address Protocol IP address Hostname Client ID or DUID
--------------------------------------------------------------------------------------------------------------------------------------------------------
2024-09-27 10:14:20 52:54:00:0d:a6:7b ipv4 192.168.122.10/24 nomad-vm-795c791f ff:08:24:45:0e:00:02:00:00:ab:11:d5:ae:0a:38:57:5f:a4:96

root@libvirt-0:/nomad-test# ssh ubuntu@192.168.122.10
Welcome to Ubuntu 20.04.6 LTS (GNU/Linux 5.4.0-193-generic x86_64)

* Documentation: https://help.ubuntu.com
* Management: https://landscape.canonical.com
* Support: https://ubuntu.com/pro

System information as of Fri Sep 27 09:18:42 UTC 2024

System load: 0.02 Processes: 125
Usage of /: 70.5% of 1.96GB Users logged in: 0
Memory usage: 0% IPv4 address for enp0s5: 192.168.122.10
Swap usage: 0%

Expanded Security Maintenance for Applications is not enabled.

0 updates can be applied immediately.

Enable ESM Apps to receive additional future security updates.
See https://ubuntu.com/esm or run: sudo pro status

The list of available updates is more than a week old.
To check for new updates run: sudo apt update

To run a command as administrator (user "root"), use "sudo ".
See "man sudo_root" for details.
```

Kernel logs showing iptables logging entries for attempting the local client ssh connection:
```console
2024-09-27T09:31:05.852474+00:00 libvirt-0 kernel: IN=lo OUT= MAC=00:00:00:00:00:00:00:00:00:00:00:00:08:00 SRC=10.0.1.17 DST=10.0.1.17 LEN=60 TOS=0x10 PREC=0x00 TTL=64 ID=48746 DF PROTO=TCP SPT=59862 DPT=22982 WINDOW=33280 RES=0x00 SYN URGP=0
2024-09-27T09:31:05.852486+00:00 libvirt-0 kernel: IN=lo OUT= MAC=00:00:00:00:00:00:00:00:00:00:00:00:08:00 SRC=10.0.1.17 DST=10.0.1.17 LEN=40 TOS=0x10 PREC=0x00 TTL=64 ID=0 DF PROTO=TCP SPT=22982 DPT=59862 WINDOW=0 RES=0x00 ACK RST URGP=0
2024-09-27T09:31:06.896442+00:00 libvirt-0 kernel: IN=lo OUT= MAC=00:00:00:00:00:00:00:00:00:00:00:00:08:00 SRC=10.0.1.17 DST=10.0.1.17 LEN=60 TOS=0x10 PREC=0x00 TTL=64 ID=35825 DF PROTO=TCP SPT=59866 DPT=22982 WINDOW=33280 RES=0x00 SYN URGP=0
2024-09-27T09:31:06.896454+00:00 libvirt-0 kernel: IN=lo OUT= MAC=00:00:00:00:00:00:00:00:00:00:00:00:08:00 SRC=10.0.1.17 DST=10.0.1.17 LEN=40 TOS=0x10 PREC=0x00 TTL=64 ID=0 DF PROTO=TCP SPT=22982 DPT=59866 WINDOW=0 RES=0x00 ACK RST URGP=0
2024-09-27T09:32:35.909451+00:00 libvirt-0 kernel: IN=lo OUT= MAC=00:00:00:00:00:00:00:00:00:00:00:00:08:00 SRC=10.0.1.17 DST=10.0.1.17 LEN=60 TOS=0x10 PREC=0x00 TTL=64 ID=46814 DF PROTO=TCP SPT=53884 DPT=22982 WINDOW=33280 RES=0x00 SYN URGP=0
2024-09-27T09:32:35.909462+00:00 libvirt-0 kernel: IN=lo OUT= MAC=00:00:00:00:00:00:00:00:00:00:00:00:08:00 SRC=10.0.1.17 DST=10.0.1.17 LEN=40 TOS=0x10 PREC=0x00 TTL=64 ID=0 DF PROTO=TCP SPT=22982 DPT=53884 WINDOW=0 RES=0x00 ACK RST URGP=0
```

Contributor guide

No contributing guide indexed for this repository

Research direction

Reproduce the failure with `nomad service info`, the shown SSH command, and the kernel iptables logs; compare it with the working `virsh console` and libvirt DHCP address paths. Trace the driver’s service-port forwarding entry point, which is not identified in the issue, and verify that connections from the local client host to the VM service address succeed without regressing external forwarding.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, linux
Domain
networking
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.