Cannot make traffic from host on Mac to guest IP with vpnkit

I could not find any e-mailing list or another forum I could post this question to so I am creating an issue which I will close eventually.

So I am trying to make inbound connect to the OSv unikernel guest running on hyperkit over vpnkit and it does not seem to work or I simply do not know which host IP address to use.

I am using Docker Desktop community version 2.0.0.4. By default I have 2 processes related to Docker/hyperkit/vpnkit running like so:

ps -ef | grep hyperkit
  501  3258  3256   0 Thu09AM ??         0:25.76 com.docker.vpnkit --ethernet fd:3 --port vpnkit.port.sock --port hyperkit://:62373/./vms/0 --diagnostics fd:4 --pcap fd:5 --vsock-path vms/0/connect --host-names host.docker.internal,docker.for.mac.host.internal,docker.for.mac.localhost --gateway-names gateway.docker.internal,docker.for.mac.gateway.internal,docker.for.mac.http.internal --vm-names docker-for-desktop --listen-backlog 32 --mtu 1500 --allowed-bind-addresses 0.0.0.0 --http /Users/****/Library/Group Containers/group.com.docker/http_proxy.json --dhcp /Users/****/Library/Group Containers/group.com.docker/dhcp.json --port-max-idle-time 300 --max-connections 2000 --gateway-ip 192.168.65.1 --host-ip 192.168.65.2 --lowest-ip 192.168.65.3 --highest-ip 192.168.65.254 --log-destination asl --udpv4-forwards 123:127.0.0.1:53859 --gc-compact-interval 1800
  501  3265  3260   0 Thu09AM ??        22:45.92 com.docker.hyperkit -A -u -F vms/0/hyperkit.pid -c 4 -m 2048M -s 0:0,hostbridge -s 31,lpc -s 1:0,virtio-vpnkit,path=vpnkit.eth.sock,uuid=fb0a6ebd-8e87-42c2-ae06-83c1027f0c55 -U 8354e93d-6601-4091-923f-0428d28d1c0c -s 2:0,ahci-hd,/Users/****/Library/Containers/com.docker.docker/Data/com.docker.driver.amd64-linux/Docker.raw -s 3,virtio-sock,guest_cid=3,path=vms/0,guest_forwards=2376;1525 -s 4,ahci-cd,/Applications/Docker.app/Contents/Resources/linuxkit/docker-for-mac.iso -s 5,ahci-cd,vms/0/config.iso -s 6,ahci-cd,/Applications/Docker.app/Contents/Resources/linuxkit/docker.iso -s 7,virtio-rnd -l com1,autopty=vms/0/tty,asl -f bootrom,/Applications/Docker.app/Contents/Resources/uefi/UEFI.fd,,

When I start OSv on hyperkit executing to virtio (instead of vpnkit), the OSv starts and grabs 192.168.64.16 (using dhcp) like so:

Invoking HYPERKIT at: /usr/local/bin/hyperkit with arguments:
  -A
  -x
  -c 2
  -m 1024M
  -f kexec,/Users/***/.capstan/repository/osv-loader/osv-vmlinuz.bin,,/usr/lib/jvm/java/bin/java -Xms64m -Xmx64m -jar spring-boot-2-rest-service-basic.jar
  -l com1,stdio
  -s 0:0,hostbridge
  -s 31,lpc
  -s 1:0,virtio-blk,/Users/***/.capstan/repository/spring-boot/spring-boot.hkit
  -s 2:0,virtio-net
OSv v0.53.0-61-g513e776b
eth0: 192.168.64.16
....

and I can make traffic over port 8080 from the host, more specifically the guest responds on HTTP at http://192.168.64.16:8080/

The extra hyperkit process looks like this:

0 19871 19869   0  6:08PM ttys007    0:39.01 /usr/local/bin/hyperkit -A -x -c 2 -m 1024M -f kexec,/Users/****/.capstan/repository/osv-loader/osv-vmlinuz.bin,,/usr/lib/jvm/java/bin/java -Xms64m -Xmx64m -jar spring-boot-2-rest-service-basic.jar -l com1,stdio -s 0:0,hostbridge -s 31,lpc -s 1:0,virtio-blk,/Users/****/.capstan/repository/spring-boot/spring-boot.hkit -s 2:0,virtio-net

Now when I use vpnkit instead I can see guest grabbing an IP but I cannot connect to it in any way and it does not work.

Invoking HYPERKIT at: /usr/local/bin/hyperkit with arguments:
  -A
  -x
  -c 2
  -m 1024M
  -f kexec,/Users/wkozaczuk/.capstan/repository/osv-loader/osv-vmlinuz.bin,,/usr/lib/jvm/java/bin/java -Xms64m -Xmx64m -jar spring-boot-2-rest-service-basic.jar
  -l com1,stdio
  -s 0:0,hostbridge
  -s 31,lpc
  -s 1:0,virtio-blk,/Users/***/.capstan/repository/spring-boot/spring-boot.hkit
  -s 2:0,virtio-vpnkit,path=/Users/****/Library/Containers/com.docker.docker/Data/vpnkit.eth.sock
  -s 3,virtio-sock,guest_cid=3,path=/Users/***/.capstan/instances/hkit/spring-boot/vsockState
OSv v0.53.0-61-g513e776b
eth0: 192.168.65.24
...

And I see this extra process:

501 19935 19933   0  6:14PM ttys007    0:30.10 /usr/local/bin/hyperkit -A -x -c 2 -m 1024M -f kexec,/Users/***/.capstan/repository/osv-loader/osv-vmlinuz.bin,,/usr/lib/jvm/java/bin/java -Xms64m -Xmx64m -jar spring-boot-2-rest-service-basic.jar -l com1,stdio -s 0:0,hostbridge -s 31,lpc -s 1:0,virtio-blk,/Users/***/.capstan/repository/spring-boot/spring-boot.hkit -s 2:0,virtio-vpnkit,path=/Users/***/Library/Containers/com.docker.docker/Data/vpnkit.eth.sock -s 3,virtio-sock,guest_cid=3,path=/Users/***/.capstan/instances/hkit/spring-boot/vsockState

I have tried obvious 192.168.65.24 as the guest reports grabbing, I have tried host IP 192.168.65.2 and couple of other ones? I event have tried to add ,guest_forwards=8080 to vsock path and also does not work with any IPs.

What am I doing wrong? Please advise.

Waldek

Start by reproducing the two hyperkit configurations and comparing the virtio-net and virtio-vpnkit paths, including the gateway and host addresses shown in the vpnkit command line. Inspect the vpnkit diagnostics and forwarding behavior for the port 8080 service. Done should identify a reachable address or a documented forwarding configuration for inbound host traffic.