Enable TLS certificate validation by default for SMTP/IMAP/FTP/POP/NNTP protocols

Feature or enhancement

I was surprised that Python does not verify hostnames by default for the stdlib modules for SMTP, IMAP, FTP, POP and NNTP. I believe the "insecure by default" behavior is no longer appropriate even for those protocol (at least SMTP and IMAP, I'm not too familiar with the rest - but even there, I suppose if an user asks for a secure connection, it should be secure).

In PEP 476 (Enabling certificate verification by default for stdlib http clients, 2014), certificate verification was enabled by default for HTTPS, with the rationale that:

The failure to do these checks means that anyone with a privileged network position is able to trivially execute a man in the middle attack against a Python application using either of these HTTP clients, and change traffic at will.

and

The “S” in “HTTPS” stands for secure. When Python’s users type “HTTPS” they are expecting a secure connection, and Python should adhere to a reasonable standard of care in delivering this. Currently we are failing at this, and in doing so, APIs which appear simple are misleading users.

When asked, many Python users state that they were not aware that Python failed to perform these validations, and are shocked.

and

The failure of various applications to note Python’s negligence in this matter is a source of regular CVE assignment

That PEP only improved that situation for HTTPS, stating that:

This PEP only proposes requiring this level of validation for HTTP clients, not for other protocols such as SMTP.

This is because while a high percentage of HTTPS servers have correct certificates, as a result of the validation performed by browsers, for other protocols self-signed or otherwise incorrect certificates are far more common. Note that for SMTP at least, this appears to be changing and should be reviewed for a potential similar PEP in the future:

https://www.facebook.com/notes/protect-the-graph/the-current-state-of-smtp-starttls-deployment/1453015901605223
https://www.facebook.com/notes/protect-the-graph/massive-growth-in-smtp-starttls-deployment/1491049534468526

Unfortunately, it seems to be very difficult to find more recent data about how many SMTP/IMAP/... servers in the wild present an invalid certificate - all I could find is that according to Google, adoption of outgoing encryption in general went up from ~75% when the PEP was written to ~90% now, and inbound encryption went up from ~57% to ~87%.

However, there seems to be a strong consensus to treat this kind of thing as a vulnerability in clients, even as early as 2007. Some examples:

• CWE - CWE-297: Improper Validation of Certificate with Host Mismatch (4.6)
• CVE - CVE-2007-5770: The (1) Net::ftptls, (2) Net::telnets, (3) Net::imap, (4) Net::pop, and (5) Net::smtp libraries in Ruby 1.8.5 and 1.8.6 do not verify that the commonName (CN) field in a server certificate matches the domain name in a request sent over SSL, which makes it easier for remote attackers to intercept SSL transmissions via a man-in-the-middle attack or spoofed web site, different components than CVE-2007-5162.
• NVD - CVE-2009-3766: mutt_ssl.c in mutt 1.5.16 and other versions before 1.5.19, when OpenSSL is used, does not verify the domain name in the subject's Common Name (CN) field of an X.509 certificate, which allows man-in-the-middle attackers to spoof SSL servers via an arbitrary valid certificate.
• CVE - CVE-2011-4318: Dovecot 2.0.x before 2.0.16, when ssl or starttls is enabled and hostname is used to define the proxy destination, does not verify that the server hostname matches a domain name in the subject's Common Name (CN) of the X.509 certificate, which allows man-in-the-middle attackers to spoof SSL servers via a valid certificate for a different hostname.
• CVE - CVE-2011-1429: Mutt does not verify that the smtps server hostname matches the domain name of the subject of an X.509 certificate, which allows man-in-the-middle attackers to spoof an SSL SMTP server via an arbitrary certificate, a different vulnerability than CVE-2009-3766.
• CVE - CVE-2012-2993: Microsoft Windows Phone 7 does not verify the domain name in the subject's Common Name (CN) field of an X.509 certificate, which allows man-in-the-middle attackers to spoof an SSL server for the (1) POP3, (2) IMAP, or (3) SMTP protocol via an arbitrary valid certificate.
• CVE - CVE-2013-0308: The imap-send command in GIT before 1.8.1.4 does not verify that the server hostname matches a domain name in the subject's Common Name (CN) or subjectAltName field of the X.509 certificate, which allows man-in-the-middle attackers to spoof SSL servers via an arbitrary valid certificate.
• CVE - CVE-2014-7273 / CVE - CVE-2014-7274: The IMAP-over-SSL implementation in getmail 4.0.0 through 4.43.0 does not verify X.509 certificates from SSL servers, which allows man-in-the-middle attackers to spoof IMAP servers and obtain sensitive information via a crafted certificate.

and more recently:

• CVE - CVE-2020-1758: A flaw was found in Keycloak [...], where it does not perform the TLS hostname verification while sending emails using the SMTP server. [...]
• CVE - CVE-2020-9488: Improper validation of certificate with host mismatch in Apache Log4j SMTP appender. [...]
• CVE - CVE-2020-13163: em-imap 0.5 uses the library eventmachine in an insecure way [...]. The hostname in a TLS server certificate is not verified.
• NVD - CVE-2020-15047: MSA/SMTP.cpp in Trojita [...] ignores certificate-verification errors, which allows man-in-the-middle attackers to spoof SMTP servers.
• https://github.com/OfflineIMAP/offlineimap/issues/669
• NVD - CVE-2021-44549: Apache Sling Commons Messaging Mail provides a simple layer on top of JavaMail/Jakarta Mail for OSGi to send mails via SMTPS. To reduce the risk of "man in the middle" attacks additional server identity checks must be performed when accessing mail servers. For compatibility reasons these additional checks are disabled by default in JavaMail/Jakarta Mail. The SimpleMailService in Apache Sling Commons Messaging Mail 1.0 lacks an option to enable these checks for the shared mail session. A user could enable these checks nevertheless by accessing the session via the message created by SimpleMessageBuilder and setting the property mail.smtps.ssl.checkserveridentity to true. Apache Sling Commons Messaging Mail 2.0 adds support for enabling server identity checks and these checks are enabled by default. (quoted in full as probably similar backwards compat concerns like in Python)

Previous discussion

• PEP 476, see above
• My Tweet, where @vstinner encouraged me to open an issue (and perhaps later PR) about this
• #72209 (2016), where @tiran proposed: make ftplib, imaplib, nntplib, pop3lib, smtplib etc. validate certs by default., but @vstinner objected (pointing to the concerns in PEP 476), and so did @ncoghlan. In August 2021, @kousu pointed out that this is still the case.

Linked PRs

• gh-91875
• gh-154712

先檢視連結的 PR gh-91875 和 gh-154712，然後檢查 issue 中提到的標準函式庫模組：smtplib、imaplib、ftplib、nntplib 和 poplib。當各通訊協定預設一致地驗證 TLS 憑證，且相容性與範圍方面的決策獲得解決時，工作即告完成。