--ignore-errors flag not working...
- Dominant language
- Python
- Stars
- 7.5k
- Forks
- 424
- PR merge metrics
- No merged PRs in 30d
Description
Hi, just discovered this and installed today.
I have a relatively large and diverse repo that has notebooks and python scripts, using a pyenv managed virtualenv with python v3.11.11. Looking to add requirements file to make it easier to work on this project across multiple systems now that I need more compute capabilities than my laptop has.
Anway, have pipreqs 0.4.13 installed, here's my experience with it...
```
$ cd ~
$ pyenv shell 3.11
$ pip install pipreqs
$ pipreqs --scan-notebooks ./project
Traceback (most recent call last):
File "/home/sa/.pyenv/versions/3.11.11/lib/python3.11/site-packages/nbformat/reader.py", line 19, in parse_json
nb_dict = json.loads(s, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^
File "/home/sa/.pyenv/versions/3.11.11/lib/python3.11/json/__init__.py", line 346, in loads
return _default_decoder.decode(s)
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/sa/.pyenv/versions/3.11.11/lib/python3.11/json/decoder.py", line 337, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/sa/.pyenv/versions/3.11.11/lib/python3.11/json/decoder.py", line 355, in raw_decode
raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/home/sa/.pyenv/versions/3.11.11/bin/pipreqs", line 8, in
sys.exit(main())
^^^^^^
File "/home/sa/.pyenv/versions/3.11.11/lib/python3.11/site-packages/pipreqs/pipreqs.py", line 609, in main
init(args)
File "/home/sa/.pyenv/versions/3.11.11/lib/python3.11/site-packages/pipreqs/pipreqs.py", line 533, in init
candidates = get_all_imports(
^^^^^^^^^^^^^^^^
File "/home/sa/.pyenv/versions/3.11.11/lib/python3.11/site-packages/pipreqs/pipreqs.py", line 136, in get_all_imports
contents = read_file_content(file_name, encoding)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/sa/.pyenv/versions/3.11.11/lib/python3.11/site-packages/pipreqs/pipreqs.py", line 183, in read_file_content
contents = ipynb_2_py(file_name, encoding=encoding)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/sa/.pyenv/versions/3.11.11/lib/python3.11/site-packages/pipreqs/pipreqs.py", line 203, in ipynb_2_py
(body, _) = exporter.from_filename(file_name)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/sa/.pyenv/versions/3.11.11/lib/python3.11/site-packages/nbconvert/exporters/templateexporter.py", line 390, in from_filename
return super().from_filename(filename, resources, **kw) # type:ignore[return-value]
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/sa/.pyenv/versions/3.11.11/lib/python3.11/site-packages/nbconvert/exporters/exporter.py", line 201, in from_filename
return self.from_file(f, resources=resources, **kw)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/sa/.pyenv/versions/3.11.11/lib/python3.11/site-packages/nbconvert/exporters/templateexporter.py", line 396, in from_file
return super().from_file(file_stream, resources, **kw) # type:ignore[return-value]
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/sa/.pyenv/versions/3.11.11/lib/python3.11/site-packages/nbconvert/exporters/exporter.py", line 221, in from_file
nbformat.read(file_stream, as_version=4), resources=resources, **kw
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/sa/.pyenv/versions/3.11.11/lib/python3.11/site-packages/nbformat/__init__.py", line 174, in read
return reads(buf, as_version, capture_validation_error, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/sa/.pyenv/versions/3.11.11/lib/python3.11/site-packages/nbformat/__init__.py", line 92, in reads
nb = reader.reads(s, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/sa/.pyenv/versions/3.11.11/lib/python3.11/site-packages/nbformat/reader.py", line 75, in reads
nb_dict = parse_json(s, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^
File "/home/sa/.pyenv/versions/4.11.11/lib/python3.11/site-packages/nbformat/reader.py", line 25, in parse_json
raise NotJSONError(message) from e
nbformat.reader.NotJSONError: Notebook does not appear to be JSON: ''
$ pipreqs --scan-notebooks --ignore-errors ./project
Usage:
pipreqs [options] []
```
Not really sure what's going on with the JSON error, don't have time to dig into it. All of the notebooks process just fine, save a few works-in-progress that have python code errors in them, but Jupyter processes the actual notebook files normally. I did run with the `--debug` flag, but there is no useful information in the output, not even a clue as to what notebook is causing the errors. Anyway, I'm fine with not catching everything, so I tried the `--ignore-errors` flag and that didn't work either.
I saw the `--ignore-errors` feature seemed to have been merged recently and it doesn't appear that there are any unit tests for it, so I'll move on to some other method to pull dependencies out of my scripts and notebooks, but wanted to file an issue to see if I could help. Happy to follow or circle back to help with this issue in a week or two.
Contributor guide
Research direction
Start at pipreqs/pipreqs.py, especially main/init, get_all_imports, read_file_content, and ipynb_2_py, and trace how --scan-notebooks and --ignore-errors are parsed and propagated. Reproduce the failure with a malformed or empty notebook, add coverage for the flag, and verify pipreqs completes without aborting the scan.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- jupyter-notebook, python
- Domain
- cli
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100