Unicode Decode Error
- Dominant language
- Python
- Stars
- 7.5k
- Forks
- 424
- PR merge metrics
- No merged PRs in 30d
Description
### Problem
In Python 3.12 running pipreqs returns this error:
```
:165: SyntaxWarning: invalid escape sequence '\S'
:166: SyntaxWarning: invalid escape sequence '\['
:207: SyntaxWarning: invalid escape sequence '\['
:456: SyntaxWarning: invalid escape sequence '\S'
:200: SyntaxWarning: invalid escape sequence '\:'
Traceback (most recent call last):
File "/*******/.local/bin/pipreqs", line 8, in
sys.exit(main())
^^^^^^
File "/*******/.local/lib/python3.12/site-packages/pipreqs/pipreqs.py", line 609, in main
init(args)
File "/*******/.local/lib/python3.12/site-packages/pipreqs/pipreqs.py", line 533, in init
candidates = get_all_imports(
^^^^^^^^^^^^^^^^
File "/*******/.local/lib/python3.12/site-packages/pipreqs/pipreqs.py", line 136, in get_all_imports
contents = read_file_content(file_name, encoding)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/*******/.local/lib/python3.12/site-packages/pipreqs/pipreqs.py", line 181, in read_file_content
contents = f.read()
^^^^^^^^
File "", line 322, in decode
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xb1 in position 81: invalid start byte
```
Therefore, it doesn't save any requirements.txt
### Possible Solution
It seems that changing the afformentioned piece of code into this one can manage to solve it temporarily:
```
with open(file_name, "r", encoding=encoding) as f:
try:
contents = f.read()
except UnicodeDecodeError:
contents = f.read().enode("utf-8")
```
Although changing this solves the main issue about saving requirements.txt file, it adds raising of a number of warnings about the name and need to check module/lib names from PyPi which slows down pipreqs.
_PS: Other changes returned similar warnings_
Contributor guide
Research direction
Reproduce the failure with Python 3.12, then inspect pipreqs.py in read_file_content and its callers get_all_imports and init. Check how the reported non-UTF-8 input should be handled without introducing the warnings described in the issue; done means pipreqs completes and saves requirements.txt for the affected project.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- cli
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 42/100