Investigate what to do with malformed characters
- Dominant language
- Rust
- Stars
- 22.3k
- Forks
- 1.9k
- PR merge metrics
- No merged PRs in 30d
Description
`Sedlex.Utf8.from_string` will throw a `MalFormed` exception if given malformed unicode characters, like `\xa0`, which should be `\xc2\xa0` in UTF8 ([source](http://www.fileformat.info/info/unicode/char/00a0/index.htm)).
@bhamiltoncx made the following suggestion:
> There are three things Flow can do:
> 1) Strict (good default): raise an error when reading file from disk
> 2) Substitute: replace non-UTF-8 sequences with U+FFFD
> 3) Add option to specify an encoding and convert files from that encoding to Unicode when reading from disk
At the moment, we will just treat the malformed file as empty. The main problem with this is that it may be a `// @flow` file and in that case we would want to emit a parsing error.
Contributor guide
Assessment
This issue has not been assessed yet.