facebook / facebook/flow

flow generates invalid utf-8 output in JSON

Open
#7,690 1 comment 0 reactions 0 assignees View on GitHub
bug
Dominant language
Rust
Stars
22.3k
Forks
1.9k
PR merge metrics
No merged PRs in 30d

Description

Flow version: 0.98.0 and prior versions

## Expected behavior

When flow outputs UTF-8 characters in JSON results, it should
contain valid UTF-8 characters.

## Actual behavior

Flow, can generate invalid UTF-8 characters on occasion in the
JSON output.

This behavior occurs when Flow truncates output with '...'
constructs.

What happens is that Flow correctly renders utf-8 output,
but then truncates the output at a _byte_ boundary instead
of a _character_ boundary when truncating output with '...'

This truncation happens in the middle of a UTF-8 character,
and so causes it's JSON output (which requires UTF-8) to be
invalid.

To discover this issue, you need to have
1) input that has encoded UTF-8 characters that is long enough to
be truncated
2) an analysis which causes that truncation
3) process the JSON output with a tool which validates JSON,
such as jsonlint.

I am not allowed to generate a minimal example to show this,
however, I can tell you how to reproduce it.

1) get current version of blockly & unpack it
wget https://github.com/google/blockly/archive/1.20190419.0.tar.gz
tar xfz 1.20190419.0.tar.gz

2) configure blockly's dependencies
cd blockly-1.20190419.0

2) Install a simple .flowconfig
cd blockly-1.20190419.0
cat <.flowconfig
[include]

[libs]

[options]

[ignore]
/node_modules
/.*\.pyc
/\.settings
/local_build\/local_.*_compressed\.js
/npm\-debug\.log
/\/nbproject\/private\/
/blockly_node_javascript_en\.js
/tests\/compile\/.*compiler.*\.jar
/\.project
/.*\.komodoproject
/local_build\/.*compiler.*\.jar
/node_modules
/\.DS_Store
/tests\/compile\/main_compressed\.js
EOF

3) run flow on blockly
flow check --weak --all --show-all-errors --json >../blockly.json

4) run a json validation tool on the resulting json.
You need to use the python3 based jsonlint using demjson
to have a "high quality" json linter which finds the error.
[ jsonlint is jsonlni on redhat systems ]
[ jsonlint is jsonlint-py3 on ubuntu like systems ]
jsonlint -f ../blockly.json >../linted.json

5) Please read the error message in the comment at the top
of the resulting json file showing the UTF-8 error.
cat ../linted.json
../blockly.json:0:0: Fatal: a Unicode decoding error occurred
| Cause: 'utf8' codec can't decode byte 0xc3 in position 11743891: invalid continuation byte
| At start of input
../blockly.json:0:0: Fatal: Aborting, can not read JSON document.
| At start of input
../blockly.json: has errors

If you want to see the source code line which does this, it is in
demos/plane/soy/soyutils.js
around line 948, there are several strings with embedded utf-8
in them. Some of the strings are long enough they are truncated
in the output of flow, right in the middle of a utf-8 character, as mentioned
at the start of this bug report.

The string in particular causing the error is bidiLtrChars.

The flow error message in the corrupted json looks like:
"message":[
{
"context":"soyshim.$$bidiLtrChars_ =",
"descr":"Cannot assign `'A-Za-zÃ<80>-Ã...' + 'â<80><8e>â°<80>-ï¬<.'` to `soyshim.$$bidiLtrChars_` because property `$$bidiLtrChars_` is missing in object literal [1].",
...

If you can fix the truncated UTF strings in '...' segments to avoid truncating a
utf-8 character in the middle of a character, it looks like it will generate valid
json.
,

* Link to Try-Flow or Github repo:

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.