Packer SSH communicator `ssh_port` argument value is ignored

#### Overview of the Issue

When specifying a value for the argument `ssh_port`, the argument value is ignored, and instead the port is chosen from the range specified by `host_port_min` and `host_port_max` is selected. I have repeated this multiple times to verify the behavior.

#### Reproduction Steps

`packer build`

### Plugin and Packer version

1.8.3 and 1.0.4

### Simplified Packer Buildfile

```hcl
source "virtualbox-vm" "ubuntu" {
ssh_host = "127.0.0.1"
ssh_port = "2222"
}
```

### Operating system and Environment details

n/a

### Log Fragments and crash.log files

> 2022/09/08 15:19:29 packer-plugin-virtualbox_v1.0.4_x5.0_linux_amd64 plugin: 2022/09/08 15:19:29 [INFO] Attempting SSH connection to 127.0.0.1:3171...
2022/09/08 15:19:29 packer-plugin-virtualbox_v1.0.4_x5.0_linux_amd64 plugin: 2022/09/08 15:19:29 [DEBUG] reconnecting to TCP connection for SSH
2022/09/08 15:19:29 packer-plugin-virtualbox_v1.0.4_x5.0_linux_amd64 plugin: 2022/09/08 15:19:29 [DEBUG] handshaking with SSH
2022/09/08 15:19:29 packer-plugin-virtualbox_v1.0.4_x5.0_linux_amd64 plugin: 2022/09/08 15:19:29 [DEBUG] SSH handshake err: ssh: handshake failed: read tcp 127.0.0.1:42718->127.0.0.1:3171: read: connection reset by peer