bndr / bndr/pipreqs

Unicode Decode Error

Open
#469 8 comments 3 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
7.5k
Forks
424
PR merge metrics
No merged PRs in 30d

Description

### Problem

In Python 3.12 running pipreqs returns this error:
```
:165: SyntaxWarning: invalid escape sequence '\S'
:166: SyntaxWarning: invalid escape sequence '\['
:207: SyntaxWarning: invalid escape sequence '\['
:456: SyntaxWarning: invalid escape sequence '\S'
:200: SyntaxWarning: invalid escape sequence '\:'
Traceback (most recent call last):
File "/*******/.local/bin/pipreqs", line 8, in
sys.exit(main())
^^^^^^
File "/*******/.local/lib/python3.12/site-packages/pipreqs/pipreqs.py", line 609, in main
init(args)
File "/*******/.local/lib/python3.12/site-packages/pipreqs/pipreqs.py", line 533, in init
candidates = get_all_imports(
^^^^^^^^^^^^^^^^
File "/*******/.local/lib/python3.12/site-packages/pipreqs/pipreqs.py", line 136, in get_all_imports
contents = read_file_content(file_name, encoding)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/*******/.local/lib/python3.12/site-packages/pipreqs/pipreqs.py", line 181, in read_file_content
contents = f.read()
^^^^^^^^
File "", line 322, in decode
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xb1 in position 81: invalid start byte
```
Therefore, it doesn't save any requirements.txt

### Possible Solution
It seems that changing the afformentioned piece of code into this one can manage to solve it temporarily:

```
with open(file_name, "r", encoding=encoding) as f:
try:
contents = f.read()
except UnicodeDecodeError:
contents = f.read().enode("utf-8")
```

Although changing this solves the main issue about saving requirements.txt file, it adds raising of a number of warnings about the name and need to check module/lib names from PyPi which slows down pipreqs.

_PS: Other changes returned similar warnings_

Contributor guide

Open the contributing guide

Research direction

Reproduce the failure with Python 3.12, then inspect pipreqs.py in read_file_content and its callers get_all_imports and init. Check how the reported non-UTF-8 input should be handled without introducing the warnings described in the issue; done means pipreqs completes and saves requirements.txt for the affected project.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
cli
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.