aboutcode-org / aboutcode-org/python-inspector

`packages` is empty when using a non-pypi.org simple index (e.g. Sonatype Nexus)

Đang mở
#268 1 bình luận 0 reaction 0 người được giao Xem trên GitHub
Ngôn ngữ chính
Python
Star
23
Fork
28
Chỉ số merge pull request
Không có pull request nào được merge trong 30 ngày

Mô tả

Currently the metadata fetch requires the pypi.org-specific JSON API. (See also #260 )

When `PYINSP_INDEX_URL` points at a non-pypi.org simple index (Sonatype Nexus PyPI in my case), `python-inspector` resolves dependencies but the `packages` array in the output is empty - package metadata is never fetched.

For example: `PYINSP_INDEX_URL=https://my-nexus-pypi.com/repository/pypi/simple`:

```shell
root@2ae255f06ad8:/project# python-inspector -p3.13 -rrequirements.txt -olinux --json-pdt /tmp/test.json --verbose
...
retrieve package data from pypi:
retrieved package 'pkg:pypi/cx-oracle@7.3.0'
retrieved package 'pkg:pypi/redis@2.10.6'
done!

root@2ae255f06ad8:/project# jq '.packages[].name' /tmp/test.json
# (no output - packages is [])
```

Note the verbose log claims `retrieved package ...`, yet `packages` is empty - the retrieval silently failed.

Without the index (default pypi.org) it works:

```shell
root@2ae255f06ad8:/project# env -u PYINSP_INDEX_URL python-inspector -p3.13 -rrequirements.txt -olinux --json-pdt /tmp/test.json --verbose
...
root@2ae255f06ad8:/project# jq '.packages[].name' /tmp/test.json
"cx-oracle"
"redis"
```

Root cause is the `get_pypi_data_from_purl` builds the metadata URL from the index base and queries the pypi.org-specific JSON API, then returns `None` on any non-200:

https://github.com/aboutcode-org/python-inspector/blob/a841c7ce82c6087695af7fc558f5ad30fb5b1e27/src/python_inspector/package_data.py#L65-L77

With a Nexus index this resolves to `https://my-nexus-pypi.com/repository/pypi/pypi/redis/2.10.6/json`, which 404s - Nexus's PyPI format implements the simple index (PEP 503), not the `/pypi///json` API - so the function bails out and no `PackageData` is produced.

I initially assumed `--use-pypi-json-api` was the toggle for this. It doesn't help here: in its default (off) position `get_pypi_data_from_purl` still unconditionally queries the JSON API, and in the on position the help states `--index-url` is ignored (i.e. pypi.org), so it can't reach a private index either way.

I'm not sure what the correct approach would be, I feel like `--use-pypi-json-api` option should be fixed that way we download the selected sdist/wheel and parse its `PKG-INFO` for `License` / `Home-page` / `Author` / `Requires-Dist`. The simple index already provides the file list + digests, so the JSON API's `urls` are largely redundant; its unique value is the rich `info` block, which `PKG-INFO` also carries.

While searching for solution I've also stumbled upon `PEP 691` https://packaging.python.org/en/latest/specifications/simple-repository-api/#json-based-simple-api-for-python-package-indexes
Which might be better than parsing of `PKG-INFO` (`pkginfo` package exists though, so it would be easy)

Either way, would like to hear the maintainers thoughts on that. If it something that should be fixed - I would try.

Hướng dẫn đóng góp

Chưa lập chỉ mục được hướng dẫn đóng góp cho kho mã nguồn này

Đánh giá

Issue này chưa được đánh giá.

Nhận issue mới trong hộp thư của bạn

Bản tóm tắt ngắn những issue GitHub phù hợp với người mới.