Fix the decode/encode hack in `twisted.web.util.formatFailure`

	@exarkun reported
Trac ID	trac#6072
Type	enhancement
Created	2012-10-06 12:38:14Z

twisted.web.util.formatFailure uses a trick to ensure that its output is ASCII-only to avoid imposing new requirements on code calling it (ie, that such code be sure to include a correct "charset" field in the Content-Type header being used). The trick is:

result[0].decode('utf-8').encode('ascii', 'xmlcharrefreplace')

This is correct but probably not particularly efficient, therefore it would be nice if this decode/encode trick were not necessary.

There was a conversation on twisted-web about some of the details surrounding this issue that may be informative. Here are some links to the first message of the thread in each month in which it occurred (because pipermail can't thread across month boundaries):

http://twistedmatrix.com/pipermail/twisted-web/2011-November/004839.html
http://twistedmatrix.com/pipermail/twisted-web/2011-December/004846.html
http://twistedmatrix.com/pipermail/twisted-web/2012-January/004854.html

trac-id__6072 6072
type__enhancement enhancement
reporter__exarkun exarkun
priority__normal normal
milestone__ 
branch__ 
branch_author__ 
status__new new
resolution__None None
component__web web
keywords__ 
time__1349527094000000 1349527094000000
changetime__1349527094000000 1349527094000000
version__None None
owner__ 
cc__jknight

Start at twisted.web.util.formatFailure and inspect how its result is produced and consumed. Read the linked twisted-web discussion for the constraints around encoding and Content-Type handling. Done means avoiding the decode/encode trick while preserving ASCII-only output and compatibility for callers that do not provide a charset.