Detect invalid UTF-8 data at end of file when using PerlIO :encoding(utf-8)
Nobody has claimed this yet.
- Dominant language
- Perl
- Stars
- 37
- Forks
- 54
- PR merge metrics
- No merged PRs in 30d
Description
PerlIO layer :encoding(utf-8) seems to fail to report malformed data at the end of a file.
Suppose a file $fn contains valid UTF-8, except for the final character in the file. The last character in the file has an invalid UTF-8 encoding. I would like to have a warning printed to STDERR about invalid UTF-8 when reading this file, but strangely it seems not possible to achieve.
For example:
use feature qw(say);
use strict;
use warnings;
binmode STDOUT, ':utf8';
binmode STDERR, ':utf8';
my $bytes = "\x{61}\x{E5}"; # 2 bytes in iso 8859-1: aå
my $fn = 'test.txt';
open ( my $fh, '>:raw', $fn ) or die "Could not open file '$fn': $!";
print $fh $bytes;
close $fh;
now $fn contains invalid UTF-8 (the last byte). If I now try to read the file using PerlIO layer :encoding(utf-8):
my $str = '';
open ( $fh, "<:encoding(utf-8)", $fn ) or die "Could not open file '$fn': $!";
$str = do { local $/; <$fh> };
close $fh;
say "Read string: '$str'";
the output is
Read string: 'a'
Note, that there is no warning "\xE5" does not map to Unicode in this case.
However, if I read the file as bytes and then use Encode::decode() on the raw data, the warnings is printed:
open ( $fh, "<:raw", $fn ) or die "Could not open file '$fn': $!";
$raw_data = do { local $/; <$fh> };
close $fh;
my $str2 = decode( 'utf-8', $raw_data, Encode::FB_WARN | Encode::LEAVE_SRC );
# warning is printed to STDERR
Why cannot the same thing be achieved with PerlIO::encoding? Is it a bug?
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by running the issue's Perl reproduction with a file whose final byte is invalid UTF-8, then compare PerlIO :encoding(utf-8) reading with Encode::decode on raw data. Trace the PerlIO encoding behavior involved in end-of-file handling. Done means the malformed final byte is detected and the expected warning is produced, or the limitation is documented with a clear explanation.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- perl
- Domain
- internationalization
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100