splitSources.py can't handle invalid UTF-8 sequences
- Dominant language
- C++
- Stars
- 25.7k
- Forks
- 6.2k
- Avg merge
- 1d 11h
- Merged PRs (30d)
- 21
Description
## Steps to Reproduce
```bash
scripts/splitSources.py test/libsolidity/syntaxTests/string/invalid_utf8_sequence.sol
```
```
Traceback (most recent call last):
File "scripts/splitSources.py", line 51, in
lines = open(filePath, mode='r', encoding='utf8').read().splitlines()
File "/usr/lib/python3.8/codecs.py", line 322, in decode
(result, consumed) = self._buffer_decode(data, self.errors, final)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xc0 in position 36: invalid start byte
```
## Description
The command above is executed by `scripts/ASTImportTest.sh` as a part of our command line tests. It's just preprocessing before the actual test.
It's still possible to split a multi-source file even if it contains invalid UTF-8 sequences so it's better to do so and let the compiler fail down the line instead of having two different layers where things can fail in different ways.
## Environment
- Compiler version: 0.7.1-develop.2020.8.31+commit.8c8eca3e.Linux.g++
Contributor guide
Research direction
Start with scripts/splitSources.py at the file-reading line shown in the traceback, then inspect how scripts/ASTImportTest.sh invokes it for test/libsolidity/syntaxTests/string/invalid_utf8_sequence.sol. Re-run the provided command; done means preprocessing completes for the invalid-UTF-8 source so the compiler can handle the later failure.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, solidity
- Domain
- cli, testing
- Issue type
- Bug
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Stale
- Clarity
- Clearly specified
- Newbie friendliness
- 45/100