Connection refused when trying to connect to Distributed worker

Hello everyone :)
I am trying to use multiple workers with Distributed, however it doesn't work as planned. I always get a connection refused error as below:

ERROR: TaskFailedException

    nested task error: IOError: connect: connection refused (ECONNREFUSED)
    Stacktrace:
     [1] worker_from_id(pg::Distributed.ProcessGroup, i::Int64)
       @ Distributed /usr/local/julia/share/julia/stdlib/v1.7/Distributed/src/cluster.jl:1089
     [2] worker_from_id
       @ /usr/local/julia/share/julia/stdlib/v1.7/Distributed/src/cluster.jl:1086 [inlined]
     [3] #remote_do#166
       @ /usr/local/julia/share/julia/stdlib/v1.7/Distributed/src/remotecall.jl:561 [inlined]
     [4] remote_do
       @ /usr/local/julia/share/julia/stdlib/v1.7/Distributed/src/remotecall.jl:561 [inlined]
     [5] kill(manager::Distributed.SSHManager, pid::Int64, config::WorkerConfig)
       @ Distributed /usr/local/julia/share/julia/stdlib/v1.7/Distributed/src/managers.jl:673
     [6] create_worker(manager::Distributed.SSHManager, wconfig::WorkerConfig)
       @ Distributed /usr/local/julia/share/julia/stdlib/v1.7/Distributed/src/cluster.jl:600
     [7] setup_launched_worker(manager::Distributed.SSHManager, wconfig::WorkerConfig, launched_q::Vector{Int64})
       @ Distributed /usr/local/julia/share/julia/stdlib/v1.7/Distributed/src/cluster.jl:541
     [8] (::Distributed.var"#41#44"{Distributed.SSHManager, Vector{Int64}, WorkerConfig})()
       @ Distributed ./task.jl:423

    caused by: IOError: connect: connection refused (ECONNREFUSED)
    Stacktrace:
     [1] wait_connected(x::Sockets.TCPSocket)
       @ Sockets /usr/local/julia/share/julia/stdlib/v1.7/Sockets/src/Sockets.jl:532
     [2] connect
       @ /usr/local/julia/share/julia/stdlib/v1.7/Sockets/src/Sockets.jl:567 [inlined]
     [3] connect_to_worker(host::String, port::Int64)
       @ Distributed /usr/local/julia/share/julia/stdlib/v1.7/Distributed/src/managers.jl:632
     [4] connect(manager::Distributed.SSHManager, pid::Int64, config::WorkerConfig)
       @ Distributed /usr/local/julia/share/julia/stdlib/v1.7/Distributed/src/managers.jl:559
     [5] create_worker(manager::Distributed.SSHManager, wconfig::WorkerConfig)
       @ Distributed /usr/local/julia/share/julia/stdlib/v1.7/Distributed/src/cluster.jl:596
     [6] setup_launched_worker(manager::Distributed.SSHManager, wconfig::WorkerConfig, launched_q::Vector{Int64})
       @ Distributed /usr/local/julia/share/julia/stdlib/v1.7/Distributed/src/cluster.jl:541
     [7] (::Distributed.var"#41#44"{Distributed.SSHManager, Vector{Int64}, WorkerConfig})()
       @ Distributed ./task.jl:423
Stacktrace:
 [1] sync_end(c::Channel{Any})
   @ Base ./task.jl:381
 [2] macro expansion
   @ ./task.jl:400 [inlined]
 [3] addprocs_locked(manager::Distributed.SSHManager; kwargs::Base.Pairs{Symbol, Any, NTuple{4, Symbol}, NamedTuple{(:exename, :dir, :sshflags, :exeflags), Tuple{String, String, Vector{String}, Cmd}}})
   @ Distributed /usr/local/julia/share/julia/stdlib/v1.7/Distributed/src/cluster.jl:487
 [4] addprocs(manager::Distributed.SSHManager; kwargs::Base.Pairs{Symbol, Any, NTuple{4, Symbol}, NamedTuple{(:exename, :dir, :sshflags, :exeflags), Tuple{String, String, Vector{String}, Cmd}}})
   @ Distributed /usr/local/julia/share/julia/stdlib/v1.7/Distributed/src/cluster.jl:447
 [5] #addprocs#249
   @ /usr/local/julia/share/julia/stdlib/v1.7/Distributed/src/managers.jl:143 [inlined]
 [6] top-level scope
   @ REPL[4]:1

The code I am trying to use to connect is as follows:

using Distributed
keyFile = raw"~/.ssh/juliakey"
addprocs([("docker@192.168.2.220:2222 0.0.0.0:3000", 1)]; exename="/usr/local/julia/bin/julia", dir="/home/docker", sshflags=["-vvv", "-i", keyFile], exeflags=`--threads=auto`)

This error shows, that the SSH connection is working perfectly fine, even a julia instance is started on the correct port (checked that with different tools), however it can't connect. Testing it with two different machines results in this error.
In this case it doesn't matter, whether this machine hosts a docker container with julia and ssh or if I try to use it natievly on windows.
I has nothing to do with the firewall, as I disabled it for testing.

The strange thing is however, that it works when I connect to myself on localhost via ssh.

To add to the confusion, I started two docker containers with julia and ssh on my machine and tried to connect between both, with the same result. It seems, that as soon as we cross the border to another machine, it fails to connect.

I added the dockerfile, if anyone wants to test it.
Steps to reproduce:

1. generate key or use existing key (update compose file)
2. docker-compose up
3. copy key to one of the containers docker cp .\juliakey dockerjl-ssh2-1:/home/docker/.ssh/juliakey
4. add IP to known hosts ssh-keyscan -p 2222 -t rsa 192.168.2.220 >> ~/.ssh/known_hosts
5. start julia with /usr/local/julia/bin/julia
6. run julia code above in the REPL
7. receive Error message

DockerJL.zip

Thanks in advance for any help :)