[Distributed] Worker connection errors (peer x not connected to y)

I've been seeing these sometimes on master/1.8. They either mess with the execution of my code or they don't, so I'm not super sure if this error is handled properly anywhere

      From worker 6:    UNHANDLED TASK ERROR: On worker 1:
      From worker 6:    On worker 7:
      From worker 6:    peer 6 has not connected to 7
      From worker 6:    Stacktrace:
      From worker 6:      [1] error
      From worker 6:        @ ./error.jl:33
      From worker 6:      [2] wait_for_conn
      From worker 6:        @ /usr/local/julia/share/julia/stdlib/v1.8/Distributed/src/cluster.jl:200
      From worker 6:      [3] JuliaLang/julia#25
      From worker 6:        @ /usr/local/julia/share/julia/stdlib/v1.8/Distributed/src/cluster.jl:186
      From worker 6:      [4] exec_conn_func
      From worker 6:        @ /usr/local/julia/share/julia/stdlib/v1.8/Distributed/src/cluster.jl:187
      From worker 6:      [5] exec_conn_func
      From worker 6:        @ /usr/local/julia/share/julia/stdlib/v1.8/Distributed/src/cluster.jl:181
      From worker 6:      [6] #invokelatest#2
      From worker 6:        @ ./essentials.jl:731
      From worker 6:      [7] invokelatest
      From worker 6:        @ ./essentials.jl:729
      From worker 6:      [8] JuliaLang/julia#116
      From worker 6:        @ /usr/local/julia/share/julia/stdlib/v1.8/Distributed/src/process_messages.jl:278
      From worker 6:      [9] run_work_thunk
      From worker 6:        @ /usr/local/julia/share/julia/stdlib/v1.8/Distributed/src/process_messages.jl:63
      From worker 6:     [10] macro expansion
      From worker 6:        @ /usr/local/julia/share/julia/stdlib/v1.8/Distributed/src/process_messages.jl:278 [inlined]
      From worker 6:     [11] JuliaLang/julia#115
      From worker 6:        @ ./task.jl:466
      From worker 6:    Stacktrace:
      From worker 6:      [1] #remotecall_fetch#165
      From worker 6:        @ /usr/local/julia/share/julia/stdlib/v1.8/Distributed/src/remotecall.jl:465
      From worker 6:      [2] remotecall_fetch
      From worker 6:        @ /usr/local/julia/share/julia/stdlib/v1.8/Distributed/src/remotecall.jl:457
      From worker 6:      [3] #remotecall_fetch#168
      From worker 6:        @ /usr/local/julia/share/julia/stdlib/v1.8/Distributed/src/remotecall.jl:492 [inlined]
      From worker 6:      [4] remotecall_fetch
      From worker 6:        @ /usr/local/julia/share/julia/stdlib/v1.8/Distributed/src/remotecall.jl:492 [inlined]
      From worker 6:      [5] JuliaLang/julia#21
      From worker 6:        @ /usr/local/julia/share/julia/stdlib/v1.8/Distributed/src/cluster.jl:171
      From worker 6:      [6] #invokelatest#2
      From worker 6:        @ ./essentials.jl:731
      From worker 6:      [7] invokelatest
      From worker 6:        @ ./essentials.jl:729
      From worker 6:      [8] JuliaLang/julia#116
      From worker 6:        @ /usr/local/julia/share/julia/stdlib/v1.8/Distributed/src/process_messages.jl:278
      From worker 6:      [9] run_work_thunk
      From worker 6:        @ /usr/local/julia/share/julia/stdlib/v1.8/Distributed/src/process_messages.jl:63
      From worker 6:     [10] macro expansion
      From worker 6:        @ /usr/local/julia/share/julia/stdlib/v1.8/Distributed/src/process_messages.jl:278 [inlined]
      From worker 6:     [11] JuliaLang/julia#115
      From worker 6:        @ ./task.jl:466
      From worker 6:    Stacktrace:
      From worker 6:     [1] remotecall_fetch(::Function, ::Distributed.Worker, ::Int64, ::Vararg{Int64}; kwargs::Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
      From worker 6:       @ Distributed /usr/local/julia/share/julia/stdlib/v1.8/Distributed/src/remotecall.jl:465
      From worker 6:     [2] remotecall_fetch(::Function, ::Distributed.Worker, ::Int64, ::Vararg{Int64})
      From worker 6:       @ Distributed /usr/local/julia/share/julia/stdlib/v1.8/Distributed/src/remotecall.jl:457
      From worker 6:     [3] #remotecall_fetch#168
      From worker 6:       @ /usr/local/julia/share/julia/stdlib/v1.8/Distributed/src/remotecall.jl:492 [inlined]
      From worker 6:     [4] remotecall_fetch
      From worker 6:       @ /usr/local/julia/share/julia/stdlib/v1.8/Distributed/src/remotecall.jl:492 [inlined]
      From worker 6:     [5] (::Distributed.var"#20#23"{Distributed.Worker})()
      From worker 6:       @ Distributed ./task.jl:466

Start with Distributed/src/cluster.jl, especially wait_for_conn and exec_conn_func, then trace the remotecall_fetch path shown in the report. Reproduce the peer connection error on Julia 1.8 if possible and inspect existing tests for worker connection failures. Done means determining whether the error is expected and properly handled, with the behavior covered by an appropriate fix or regression test.