argotorg / argotorg/solidity

splitSources.py can't handle invalid UTF-8 sequences

Open
#9,710 6 comments 0 reactions 0 assignees View on GitHub
bug :bug: low impact medium effort nice to have testing :hammer:
Dominant language
C++
Stars
25.7k
Forks
6.2k
Avg merge
1d 11h
Merged PRs (30d)
21

Description

## Steps to Reproduce
```bash
scripts/splitSources.py test/libsolidity/syntaxTests/string/invalid_utf8_sequence.sol
```
```
Traceback (most recent call last):
File "scripts/splitSources.py", line 51, in
lines = open(filePath, mode='r', encoding='utf8').read().splitlines()
File "/usr/lib/python3.8/codecs.py", line 322, in decode
(result, consumed) = self._buffer_decode(data, self.errors, final)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xc0 in position 36: invalid start byte
```

## Description
The command above is executed by `scripts/ASTImportTest.sh` as a part of our command line tests. It's just preprocessing before the actual test.

It's still possible to split a multi-source file even if it contains invalid UTF-8 sequences so it's better to do so and let the compiler fail down the line instead of having two different layers where things can fail in different ways.

## Environment
- Compiler version: 0.7.1-develop.2020.8.31+commit.8c8eca3e.Linux.g++

Contributor guide

Open the contributing guide

Research direction

Start with scripts/splitSources.py at the file-reading line shown in the traceback, then inspect how scripts/ASTImportTest.sh invokes it for test/libsolidity/syntaxTests/string/invalid_utf8_sequence.sol. Re-run the provided command; done means preprocessing completes for the invalid-UTF-8 source so the compiler can handle the later failure.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, solidity
Domain
cli, testing
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Stale
Clarity
Clearly specified
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.