Difference between `--use-local` and normal behaviour
- Dominant language
- Python
- Stars
- 7.5k
- Forks
- 424
- PR merge metrics
- No merged PRs in 30d
Description
Hi all,
I am trying to run pipreqs on my repository I run into a problem when I use the `--use-local` flag. I've checked the code and it seems that there is a small discrepancy between the two modes when trying to get a package that is in the `mapping` file. In my example I am missing the `scikit-learn` package, which is installed as `scikit-learn` but is imported as `sklearn`. Adding to the problem is the `sklearn` package on PyPi, which is deprecated. Of course you are familiar with this.
In the normal mode, where PyPi is being queried, the candidates are mapped to their respective counterpart: `sklearn` is mapped to `scikit-learn` (or `scikit_learn`). This works in the "online" mode where PyPi is queried. This fails in the `--use-local` mode because `get_locally_installed_packages()` returns the following for this package:
```
'sklearn': {'version': '0.22', 'name': 'scikit_learn'}
```
In the following step, for the `--use-local` mode, it will check if it can find the package in the keys if this dict. But `sklearn` has been mapped to `scikit_learn` so it will not find it in the dict and it will return an empty `requirements.txt`. When I remove line 1024 from the `mapping` file, which contains `sklearn:scikit_learn`, then it works with the `--use-local` mode, but it fails with it. Without that flag, it will return a `requirements.txt` with two entries: `scikit-learn` and `sklearn`. Of course, this is unwanted behaviour.
I've make a hotfix for the `--use-local` flag, but I think the code should be aligned for both modes.
My suggestion for the hotfix is to change the behaviour of the `get_import_local()` function from:
```python
def get_import_local(imports, encoding=None):
local = get_locally_installed_packages()
result = []
for item in imports:
if item.lower() in local:
result.append(local[item.lower()])
# removing duplicates of package/version
result_unique = [
dict(t)
for t in set([
tuple(d.items()) for d in result
])
]
return result_unique
```
to
```python
def get_import_local(imports, encoding=None):
local = get_locally_installed_packages()
result = []
for item in imports:
for pkg, attributes in local.items():
if item == pkg.lower() or item == attributes['name'].lower():
result.append(local[pkg])
break
# removing duplicates of package/version
result_unique = [
dict(t)
for t in set([
tuple(d.items()) for d in result
])
]
return result_unique
```
It seems that for `sklearn` the name and the pkg name are interchangeable, I assume this to happen in more packages, therefore a check if the import is in the keys or as the name of the package makes sense but I don't know much about the inner workings of pip, so any suggestions are welcome.
Let me know what you think.
I am working on a MacBook Pro with:
```
conda 4.7.10
python 3.7.4
pipreqs 0.4.8
```
Contributor guide
Research direction
Start with get_import_local() and get_locally_installed_packages(), then inspect the mapping file entry for sklearn:scikit_learn. Reproduce the discrepancy with --use-local and the sklearn/scikit-learn imports. Done means local and online modes resolve the mapping consistently and do not produce a separate deprecated sklearn requirement.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- cli
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Clearly specified
- Newbie friendliness
- 35/100