Cannot load a PEM certificate with Unicode in subject

Nobody has claimed this yet.

	Phidica reported
Trac ID	trac#9804
Type	defect
Created	2020-04-17 15:40:27Z
Branch	https://github.com/wiml/twisted/tree/non_ascii_DNs

I'm using the Python 3.7 interface to Twisted 19.2.1 (Fedora 31).

I have the following certificate in PEM format that I'm trying to load:

(For your interest, this happens to be one of the root certificates in the system CA bundle, not some certificate I just made up.)

I do so as follows:

from twisted.internet.ssl import Certificate
with open('cert.pem') as file:
  Certificate.loadPEM( file.read() )

This returns the following error:

Traceback (most recent call last):
  File "<stdin>", line 2, in <module>
  File "/usr/lib64/python3.7/site-packages/twisted/internet/_sslverify.py", line 456, in __repr__
    self.getSubject().commonName,
  File "/usr/lib64/python3.7/site-packages/twisted/internet/_sslverify.py", line 411, in getSubject
    return self._copyName('subject')
  File "/usr/lib64/python3.7/site-packages/twisted/internet/_sslverify.py", line 400, in _copyName
    dn._copyFrom(getattr(self.original, 'get_'+suffix)())
  File "/usr/lib64/python3.7/site-packages/twisted/internet/_sslverify.py", line 331, in _copyFrom
    setattr(self, name, value)
  File "/usr/lib64/python3.7/site-packages/twisted/internet/_sslverify.py", line 355, in __setattr__
    value = value.encode("ascii")
UnicodeEncodeError: 'ascii' codec can't encode character '\u0151' in position 28: ordinal not in range(128)

It appears the problem is that X.509 name fields in the subject are forcibly encoded to 'ascii', which fails in this case because the OrganizationalUnitName and CommonName contain Unicode characters.

$ openssl x509 -in cert.pem -subject -noout
subject=C = HU, L = Budapest, O = NetLock Kft., OU = Tan\C3\BAs\C3\ADtv\C3\A1nykiad\C3\B3k (Certification Services), CN = NetLock Arany (Class Gold) F\C5\91tan\C3\BAs\C3\ADtv\C3\A1ny
$ openssl x509 -in cert.pem -subject -noout -nameopt utf8
subject=C=HU, L=Budapest, O=NetLock Kft., OU=Tanúsítványkiadók (Certification Services), CN=NetLock Arany (Class Gold) Főtanúsítvány

What confuses me the most is that what I can see and interpret of X.509 specs implies that UTF-8 is not only a valid encoding for these fields, it is recommended. So I can't really understand why these fields are forcibly encoded to (and later, decoded from) ASCII, giving rise to this issue.

Searchable metadata

trac-id__9804 9804
type__defect defect
reporter__Phidica Phidica
priority__normal normal
milestone__None None
branch__https___github_com_wiml_twisted_tree_non_ascii_DNs https://github.com/wiml/twisted/tree/non_ascii_DNs
branch_author__ 
status__assigned assigned
resolution__None None
component__core core
keywords__ssl_tls_unicode ssl tls unicode
time__1587138027873608 1587138027873608
changetime__1590807088264726 1590807088264726
version__None None
owner__wiml wiml
cc__Phidica

Start in twisted/internet/_sslverify.py, especially _copyName and DistinguishedName attribute handling, and reproduce the failure through Certificate.loadPEM using the Unicode-containing certificate in the report. Check how subject fields are copied and encoded, then verify that loading and inspecting the certificate succeeds without the reported UnicodeEncodeError.

Written by the indexing model from the issue text.

Description

Contributor guide

First steps

Research direction

Assessment