Message affected by 'use utf8', breaks binary POSTs [rt.cpan.org #77403]

Migrated from rt.cpan.org#77403 (status was 'open')

Requestors:

• gortan@cpan.org
• henrik.pauli@gmail.com

Attachments:

• test-latin1.pl
• test-utf8.pl

From henrik.pauli@gmail.com on 2012-05-24 13:30:09:

It appeared to us that POSTing binary data with LWP corrupted the data
when (and only when) we had ‘use utf8’ enabled in the script using LWP.

This bug was present in LWP 5.833 as well as the newest HTTP::Message 6.03.

‘use utf8’ doesn't do anything but turn the strings in the source code
into string of characters, rather than octets -- it seems that
HTTP::Request::Common is completely encoding (and u-string) agnostic,
which is VERY dangerous in a place where you manipulate octet streams.

The source of the problem is that you have strings in the source code
(eg. where you add the Content-Disposition header[1]), and *also* read
bytes from the file into the same buffer later on[2].  One is easily a
character string, the other is definitely an octet stream.

Not sure what the right solution is, but the module should safeguard
itself against these kinds of things.

[1]
https://metacpan.org/source/GAAS/HTTP-Message-6.03/lib/HTTP/Request/Common.pm#L135
[2]
https://metacpan.org/source/GAAS/HTTP-Message-6.03/lib/HTTP/Request/Common.pm#L243

P.S. Might be a similar issue, we also recently noticed that https and
use utf8 breaks a HTTP request, either or both of them missing doesn't.

PPS. Perl 5.10.1, Linux 3.1 x86.

From gaas@cpan.org on 2012-05-27 11:48:49:

It would be helpful if you can provide a small test script that demonstrates
the problem.

From gortan@cpan.org on 2015-05-13 15:51:32:

On Sun May 27 07:48:49 2012, GAAS wrote:
> It would be helpful if you can provide a small test script that demonstrates the problem.

I think I just ran into the same issue, and tried to come up with two minimal scripts: Both have a constant value 'öööö' in their source code, which they both pass on to HTTP::Request::Common::POST to print them as application/x-www-form-urlencoded. One of the scripts is saved as latin-1, the other is saved as utf-8 and has "use utf8" set.
I would assume that the output of both scripts is identical. However, while the latin1 script produces the expected:
text=%F6%F6%F6%F6%F6%F6%F6%F6%F6%F6%F6
the utf8 script (imho incorrectly) produces:
text=%C3%B6%C3%B6%C3%B6%C3%B6%C3%B6%C3%B6%C3%B6%C3%B6%C3%B6%C3%B6%C3%B6

$HTTP::Request::Common::VERSION is 6.04, perl v5.20.2 built for x86_64-linux.

Start with the attached test-latin1.pl and test-utf8.pl scripts and the HTTP::Request::Common POST handling referenced in the report. Reproduce the differing form-encoded output with and without "use utf8", then trace how the POST body combines source strings and binary data. Done means the intended bytes are preserved and regression coverage demonstrates the corrected behavior.