pipreqs --force fails with UnicodeDecodeError when scanning non-UTF-8 Python files
- Dominant language
- Python
- Stars
- 7.5k
- Forks
- 424
- PR merge metrics
- No merged PRs in 30d
Description
(.venv) sys-317@DESKTOP-64SE9IT:/mnt/e/mamba_trial$ pipreqs
INFO: Not scanning for jupyter notebooks.
Traceback (most recent call last):
File "/mnt/e/mamba_trial/.venv/bin/pipreqs", line 7, in
sys.exit(main())
File "/mnt/e/mamba_trial/.venv/lib/python3.10/site-packages/pipreqs/pipreqs.py", line 609, in main
init(args)
File "/mnt/e/mamba_trial/.venv/lib/python3.10/site-packages/pipreqs/pipreqs.py", line 533, in init
candidates = get_all_imports(
File "/mnt/e/mamba_trial/.venv/lib/python3.10/site-packages/pipreqs/pipreqs.py", line 136, in get_all_imports
contents = read_file_content(file_name, encoding)
File "/mnt/e/mamba_trial/.venv/lib/python3.10/site-packages/pipreqs/pipreqs.py", line 181, in read_file_content
contents = f.read()
File "/usr/lib/python3.10/codecs.py", line 322, in decode
(result, consumed) = self._buffer_decode(data, self.errors, final)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xb1 in position 81: invalid start byte
Contributor guide
Research direction
The error occurs in pipreqs/pipreqs.py at line 181 in read_file_content. Start by examining that function to see how it reads files and handles encoding. The issue is likely that the file is being read with a default UTF-8 encoding when it contains non-UTF-8 characters. Look for where encoding is set or passed. Check if there's a way to detect file encoding or fallback to a different encoding. Run pipreqs on a test directory with a non-UTF-8 Python file to reproduce the error. The fix should ensure the function can handle various encodings gracefully.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- cli, tooling
- Issue type
- Bug
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 75/100