Discussion: SSL backend

There are two main TLS libraries people use in Python: the stdlib ssl module, and the third-party pyopenssl. So far Trio has stuck with the stdlib module. @dholth recently raised the question of support pyopenssl, and at time of writing has made significant progress on the implementation: #1140. The trade-offs here are super complicated, and keep coming up in various ways, so it's probably worth having a proper thread to collect discussion.

Considerations

SSLStream is some of the most complicated code in Trio, and has some of the most complicated tests, and any bugs are potentially security-sensitive. We really don't want to have to maintain multiple copies of it.

There's no reasonable way to abstract away the difference between these libraries from users. Users have to be able to configure the openssl Context object directly, and these two libraries have different and incompatible ways of doing that. Some projects (e.g. Twisted) have tried to define their own TLS configuration API so they can hide the underlying library, but it's a huge amount of work, means users lose access to what meager documentation exists for the other systems, and will inevitably end up hiding away features that people need.

ssl is in the stdlib, so always available. pyopenssl does have solid wheel coverage, so it's not a major burden for most users to install. So I guess the only case where this would matter is if pip ever wants to depend on Trio, then we'll need to be able to run Trio without any extension modules. Currently that's how it works on Linux/macOS; only Windows needs C extensions (specifically cffi). This is probably not worth worrying about too much -- if/when it makes sense for pip to seriously consider using Trio, we can figure out our options then.

ssl is in many ways simpler and easier to use. For example, a critical feature in any TLS library is to validate hostnames, or else all this certificate infrastructure is useless. With ssl, this is enabled by default. With pyopenssl, you need to be an expert and use a third-party library to enable it. Super tricky.

pyopenssl is generally more featureful. ssl has gotten a lot more complete in recent years, so we can mostly get away with it, but historically the pattern has been that serious frameworks always have something they need that they can't get from ssl, so . That's @dholth's situation... he wants OCSP stapling, which pyopenssl supports and ssl doesn't.

pyopenssl isn't tied to the CPython release cycle. Example of why this matters: openssl has some annoying bugs in TLS 1.3 that cause our tests to deadlock (https://github.com/openssl/openssl/issues/7967). For now we're disabling TLS 1.3 in the tests. But since TLS 1.3 didn't exist when CPython 3.5 and 3.6 were released, they don't expose the flag you need to disable TLS 1.3! So if you have a CPython 3.5 or 3.6 that's built against openssl 1.1.1 w/ TLS 1.3 support, then our tests just break. In CPython 3.8 will expose some more TLS 1.3 configuration knobs and might let us work around this issue (bpo-37120)... but earlier versions never will. And in general this will also apply to any future limitations we might run into -- if we stick with ssl then we have to cross our fingers that we won't run into anything too dire in the future, because if we do then we're stuck.

• Bonus: RHEL 8 has CPython 3.6 and openssl 1.1.1, so unless we can convince the openssl devs to fix 1.1.1, we're probably going to be stuck supporting this combination for a long time.

In the long run, it would be Very Nice to support platform-native TLS libraries (SChannel on Windows, SecureTransport on macOS), since these are the only libraries that know how to hook into the system trust store to properly evaluate certificates. (rustls is also super cool.) Of course this is in direct contradiction to several of the above points. I was hoping PEP 543 would come along and save us, but it seems to be stalled for now. And at least as currently conceived, it's only trying to expose a minimal common subset, so it probably won't help with situations like @dholth's OCSP stapling.

Options

The Trio community as a whole could only support ssl (= status quo), or only support pyopenssl. I suspect that this won't be viable in the long run -- even if the core Trio package only supports one, there are enough tradeoffs between these that someone will need to support the other somehow.

We could copy/paste trio's SSLStream code and tests, then hack them to make PyOpenSSLStream and tests. (This is what #1140 does for now.) And then:

• ...ship both sets of code inside trio itself.
• ...ship ssl support in Trio itself, while relegating pyopenssl to some kind of third-party library
• ...ship pyopenssl support in Trio itself, while relegating ssl to some kind of third-party library

It's nice that it's easy to ship code to use Trio+your-favorite-TLS-library as a third-party package, but all of these options do lead to a lot of tricky duplicate code. In the latter two cases we outsource that to third-parties, but someone would still need to do the work.

We could take on making PEP 543 happen, but that's a ton of work and as mentioned above, it still doesn't help with @dholth's use case.

I guess I can imagine a different/more limited version of PEP 543, where we don't try to abstract over configuration, but just the minimal set of features that Trio needs to wire up a TLS library to its I/O model. Like, let users use whatever context object they want, as long as they tell us how to do the core encrypt/decrypt/WantRead/WantWrite/feed-the-underlying-buffer operations? I guess that could even be used as a driver to grow something like PEP 543 over time, and then eventually extract it...

A more limited version of this would be to invent a little abstraction over just ssl and pyopenssl, and use that to write SSLStream. (So the user experience would be: when you create an SSLStream, you can pass in either a ssl.Context or OpenSSL.SSL.Context, your choice.) The two libraries are very similar from our point of view, because they both expose openssl's model more-or-less directly. unwrap might need a bit of care (ssl doesn't expose SSL_shutdown directly but instead wraps some automatic retry semantics around it, while I think pyopenssl just gives you SSL_shutdown), but AFAICT everything else maps pretty-much one-to-one. The biggest API issue is all the accessors that we currently re-export:

https://github.com/python-trio/trio/blob/b9941085c38939835ea82c8d3ee987b0a8c70ed5/trio/_ssl.py#L348-L395

These are all specific to ssl. One option would be to give up and just give users access to the underlying ssl.SSLObject/OpenSSL.SSL.Connection, so e.g. instead of doing ssl_stream.getpeercert() you'd do ssl_stream.ssl_object.getpeercert(). This would also remove all that dynamic stuff that makes mypy grumpy. OTOH, it would mean that the type of SSLStream.ssl_object was statically Union[ssl.SSLObject, OpenSSL.SSL.Connection], which will also be annoying to mypy users, in a different way! Mayyybe we could do something with generics and overloads, so if you pass a ssl.Context you get a SSLStream[ssl.SSLObject], but if you pass a OpenSSL.SSL.Context you get a SSLStream[OpenSSL.SSL.Connection]?

@tiran: I don't have any specific questions but I guess you might be interested in this so I'll CC you and you can unsubscribe if you want :-)

Start with trio/_ssl.py at the referenced accessor section and review implementation progress in #1140. Compare the existing SSLStream API and tests with the proposed ssl and pyopenssl options; this discussion is done only when the project reaches and records a clear design decision.