Requests toolbelt HostHeaderSSL Adapter has issues with non-standard ports.

Summary.
I've posted this in the requests.toolbelt issues as well as urlib3 issues. My challenge is following what is happening from from requests.toolbelt all the way to urllib3 is beyond my capabilities right now and I am looking for help.

The original issue on requests toolbelt is here:

https://github.com/requests/toolbelt/issues/189

The problem is shown in completeness here:

https://github.com/JohnOmernik/jupyter_drill/blob/master/issue.md

I also opened up an issue in URLLIB3:

https://github.com/urllib3/urllib3/issues/1411

However, here I was requested to show just with urllib3 the issue with assert_hostname (what the HostHeaderSSLAdapter is using) and here, urlib3 returned as expected. However, requests is using a pool manager, and I am unclear how to utilize that at the urllib3 level (with understanding of what is happening enough to recreate the problem... my testing with urllib3 was just using the test scripts included with urllib3 that references assert_hostname.)

Expected Result

I expect that using the HostHeaderSSL Adapter, I should be able to connect to an IP address, but pass the hostname to the SSL verification allowing a sticky connection to specific IP while validating the SSL certificate.

Actual Result

When I assert a new hostname, the non-standard port is rewritten to be the https default of 443. While the certificate host is correct, the connection fails because of instead of port 20004 it connects on port 443 when using assert_hostname

Reproduction Steps

Please see above posted URLs and issues, the best way to describe this is here:

https://github.com/JohnOmernik/jupyter_drill/blob/master/issue.md

System Information

$ python -m requests.help

{
  "chardet": {
    "version": "3.0.3"
  },
  "cryptography": {
    "version": "1.8.1"
  },
  "idna": {
    "version": ""
  },
  "implementation": {
    "name": "CPython",
    "version": "3.6.1"
  },
  "platform": {
    "release": "4.4.0-127-generic",
    "system": "Linux"
  },
  "pyOpenSSL": {
    "openssl_version": "100020bf",
    "version": "17.0.0"
  },
  "requests": {
    "version": "2.19.1"
  },
  "system_ssl": {
    "version": "100020ff"
  },
  "urllib3": {
    "version": "1.23"
  },
  "using_pyopenssl": true
}

This command is only available on Requests v2.16.4 and greater. Otherwise,
please provide some basic information about your system (Python version,
operating system, &c).

Start with the linked issue.md reproduction and trace the HostHeaderSSLAdapter through the pool manager and urllib3's assert_hostname handling. Compare the Requests/toolbelt behavior with the standalone urllib3 reproduction. Done means connecting to the specified non-standard port while validating the supplied hostname.