Possible race condition in TLS handshake on windows

Hello hello! I've said this before, but it definitely bears repeating: thanks for all the hard work on Trio. It's still part of my go-to async toolbox, even after so many years and kilometers moved :)

The problem

I'm getting two different errors, both of which ultimately result in a broken resource. They have the same source call site -- await stream.do_handshake() (to force the TLS handshake so that an SNI is available). They're very unpredictable:

• I can't see any pattern as to which of the two errors gets raised
• So far, the first request always succeeds
• Sometimes, the second request already encounters an error
• Sometimes, a few tens of requests get through before finally erroring

As far as I can tell (again, these are transient errors, so I can't be 100% sure):

• It appears that if I comment out the call to do_handshake and return a static SNI hostname, everything works as expected. Or at least, with this one single change, I was able to do several hundred requests spread out over however long it took me to eat dinner
• I still saw errors after removing the .sni_callback = sni_callback line, so it doesn't appear that the callback itself is affecting things
• I still saw errors after adding trio.sleep calls before stream.do_handshake, so that probably doesn't have any effect (not that I'd expect it to)
• I still saw errors after changing the port mapping (trying to rule out some unknown weird windows thing)
• It's definitely not somehow reusing socket objects or something truly bizzare
• (there may be some other things I tried that I'm forgetting)

Could this be a race condition in the TLS handshake? Partial handshakes come to mind, but... TLS is definitely not my forte, so I have no idea if that could be relevant or not.

I've included both the source code (it's a WIP, so don't mind all of the construction mess) and screenshots of the errors (sorry... windows copy/paste from the console is garbage) in this gist. I also left in all of the machete-mode-debugging-print calls, so you can see where I instrumented while troubleshooting.

Background

In my free time I'm playing around with building myself a slick development environment, because I have tons of ideas, and I want to make it as easy as possible to execute on them. There are a lot of moving parts here, and in most cases I'd say this is completely unnecessary. But hey, it's my free time, yknow? The relevant bit is this:

• When I'm testing stuff in my browser, I want to use TLS, and I want to use normal hostnames. So for example, I want to be able to say https://fooproject.<subdomain.i.own>, https://barproject.<subdomain.i.own>, etc, and have these be routed to the appropriate docker container
• I might be running multiple projects at the same time, and they might depend on each other
• I should be able to create a new project purely by cookiecuttering templates; I don't want to be mucking about in nginx config files
• I want the destination port to be dynamic. In other words, I want each project docker container to just be able to select any open port, and then the terraform-based launching service can just read the port allocation and dynamically update the corresponding port for the project domain
• I also want to be able to access non-http resources, for example, postgres and redis, also using the same host-based routing

So the solution looks like this:

1. Run unbound on my local machine. Normally it just forwards to cloudflare's DNS, but for wildcard domains *.local.taev.dev (which I own), it has a hard-coded response for 127.0.0.1
2. create a CSR locally on my devhost for a DV wildcard cert for *.local.taev.dev
3. forward that CSR to a control plane server that relays it to let's encrypt and handles the actual domain validation challenge
4. return the signed cert back to my devhost
5. within the devhost application, I've written a trio-based reverse proxy. It terminates the TLS connection, forwarding requests via TCP to the destination docker container. (I'll also add path-based routing for HTTP, but haven't quite gotten that far yet). The host name for routing comes from the TLS SNI callback.

Step 5 is where I've run into trouble.

Start with the await stream.do_handshake() call and the Windows reproduction code in the linked gist. Compare the two reported errors across repeated requests and inspect the SNI callback and handshake behavior. Done means identifying whether Trio causes the race and adding a reliable regression test or documenting that the problem lies elsewhere.