Perl-Toolchain-Gang / Perl-Toolchain-Gang/Module-Metadata

handling encodings

Open
#2 5 comments 0 reactions 1 assignee View on GitHub

@karenetheridge is already working on this.

Since Mar 20, 2015.

enhancement
Dominant language
Perl
Stars
8
Forks
17
PR merge metrics
No merged PRs in 30d

Description

Here is my (very early in the morning) assessment about what we should do regarding scanning the file and encodings (related issue: https://rt.cpan.org/Ticket/Display.html?id=78434)

  • we need different interfaces for new-from-string, new-from-file, new-from-handle. new-from-string will assume decoded characters. new-from-file takes an encoding parameter, and uses binmode(":encoding($encoding)") on the handle. new-from-handle assumes that the caller has already set encoding layers appropriately.
  • new-from-file may not have been provided an encoding parameter. in this case we start reading assuming ascii. If we encounter a 'use utf8;' directive, we back up and start over with the correct encoding applied. We will also flag a warning (error?) if we encounter a 'use utf8' directive when we were explicitly passed a different encoding parameter than 'utf8' (UTF-8?).
  • pod can have a different encoding than the code. when starting to read pod, and we don't already have an encoding supplied (can we assume that new-from-string is decoded and therefore exempt ourself from this step?) and we encounter an '=encoding ...' directive, we decode the extracted pod lines as they are read. Test cases need to consider corpus examples where the code is in a different encoding than the (interleaved) pod (can this happen in any situation where the code is not ascii, which is utf8-compatible?)

Am I missing anything?

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.