facebook / facebook/flow

Investigate what to do with malformed characters

Open
#3,814 0 comments 1 reaction 1 assignee Claimed by @mroch View on GitHub
parsing
Dominant language
Rust
Stars
22.3k
Forks
1.9k
PR merge metrics
No merged PRs in 30d

Description

`Sedlex.Utf8.from_string` will throw a `MalFormed` exception if given malformed unicode characters, like `\xa0`, which should be `\xc2\xa0` in UTF8 ([source](http://www.fileformat.info/info/unicode/char/00a0/index.htm)).

@bhamiltoncx made the following suggestion:

> There are three things Flow can do:
> 1) Strict (good default): raise an error when reading file from disk
> 2) Substitute: replace non-UTF-8 sequences with U+FFFD
> 3) Add option to specify an encoding and convert files from that encoding to Unicode when reading from disk

At the moment, we will just treat the malformed file as empty. The main problem with this is that it may be a `// @flow` file and in that case we would want to emit a parsing error.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.