dankogai / dankogai/p5-encode

Detect invalid UTF-8 data at end of file when using PerlIO :encoding(utf-8)

Open
#59 7 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Perl
Stars
37
Forks
54
PR merge metrics
No merged PRs in 30d

Description

PerlIO layer :encoding(utf-8) seems to fail to report malformed data at the end of a file.
Suppose a file $fn contains valid UTF-8, except for the final character in the file. The last character in the file has an invalid UTF-8 encoding. I would like to have a warning printed to STDERR about invalid UTF-8 when reading this file, but strangely it seems not possible to achieve.
For example:

use feature qw(say);
use strict;
use warnings;

binmode STDOUT, ':utf8';
binmode STDERR, ':utf8';

my $bytes = "\x{61}\x{E5}";  # 2 bytes in iso 8859-1: aå
my $fn = 'test.txt';
open ( my $fh, '>:raw', $fn ) or die "Could not open file '$fn': $!";
print $fh $bytes;
close $fh;

now $fn contains invalid UTF-8 (the last byte). If I now try to read the file using PerlIO layer :encoding(utf-8):

my $str = '';
open ( $fh, "<:encoding(utf-8)", $fn ) or die "Could not open file '$fn': $!";
$str = do { local $/; <$fh> };
close $fh;
say "Read string: '$str'";

the output is

Read string: 'a'

Note, that there is no warning "\xE5" does not map to Unicode in this case.

However, if I read the file as bytes and then use Encode::decode() on the raw data, the warnings is printed:

open ( $fh, "<:raw", $fn ) or die "Could not open file '$fn': $!";
$raw_data = do { local $/; <$fh> };
close $fh;
my $str2 = decode( 'utf-8', $raw_data, Encode::FB_WARN | Encode::LEAVE_SRC );
# warning is printed to STDERR

Why cannot the same thing be achieved with PerlIO::encoding? Is it a bug?

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by running the issue's Perl reproduction with a file whose final byte is invalid UTF-8, then compare PerlIO :encoding(utf-8) reading with Encode::decode on raw data. Trace the PerlIO encoding behavior involved in end-of-file handling. Done means the malformed final byte is detected and the expected warning is produced, or the limitation is documented with a clear explanation.

Written by the indexing model from the issue text.

Assessment

Tech stack
perl
Domain
internationalization
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.