aboutcode-org / aboutcode-org/python-inspector

`packages` is empty when using a non-pypi.org simple index (e.g. Sonatype Nexus)

Aberta
#268 1 comentário 0 reações 0 responsáveis Ver no GitHub
Linguagem predominante
Python
Estrelas
23
Forks
28
Métricas de merge de PRs
Nenhum PR com merge em 30d

Descrição

Currently the metadata fetch requires the pypi.org-specific JSON API. (See also #260 )

When `PYINSP_INDEX_URL` points at a non-pypi.org simple index (Sonatype Nexus PyPI in my case), `python-inspector` resolves dependencies but the `packages` array in the output is empty - package metadata is never fetched.

For example: `PYINSP_INDEX_URL=https://my-nexus-pypi.com/repository/pypi/simple`:

```shell
root@2ae255f06ad8:/project# python-inspector -p3.13 -rrequirements.txt -olinux --json-pdt /tmp/test.json --verbose
...
retrieve package data from pypi:
retrieved package 'pkg:pypi/cx-oracle@7.3.0'
retrieved package 'pkg:pypi/redis@2.10.6'
done!

root@2ae255f06ad8:/project# jq '.packages[].name' /tmp/test.json
# (no output - packages is [])
```

Note the verbose log claims `retrieved package ...`, yet `packages` is empty - the retrieval silently failed.

Without the index (default pypi.org) it works:

```shell
root@2ae255f06ad8:/project# env -u PYINSP_INDEX_URL python-inspector -p3.13 -rrequirements.txt -olinux --json-pdt /tmp/test.json --verbose
...
root@2ae255f06ad8:/project# jq '.packages[].name' /tmp/test.json
"cx-oracle"
"redis"
```

Root cause is the `get_pypi_data_from_purl` builds the metadata URL from the index base and queries the pypi.org-specific JSON API, then returns `None` on any non-200:

https://github.com/aboutcode-org/python-inspector/blob/a841c7ce82c6087695af7fc558f5ad30fb5b1e27/src/python_inspector/package_data.py#L65-L77

With a Nexus index this resolves to `https://my-nexus-pypi.com/repository/pypi/pypi/redis/2.10.6/json`, which 404s - Nexus's PyPI format implements the simple index (PEP 503), not the `/pypi///json` API - so the function bails out and no `PackageData` is produced.

I initially assumed `--use-pypi-json-api` was the toggle for this. It doesn't help here: in its default (off) position `get_pypi_data_from_purl` still unconditionally queries the JSON API, and in the on position the help states `--index-url` is ignored (i.e. pypi.org), so it can't reach a private index either way.

I'm not sure what the correct approach would be, I feel like `--use-pypi-json-api` option should be fixed that way we download the selected sdist/wheel and parse its `PKG-INFO` for `License` / `Home-page` / `Author` / `Requires-Dist`. The simple index already provides the file list + digests, so the JSON API's `urls` are largely redundant; its unique value is the rich `info` block, which `PKG-INFO` also carries.

While searching for solution I've also stumbled upon `PEP 691` https://packaging.python.org/en/latest/specifications/simple-repository-api/#json-based-simple-api-for-python-package-indexes
Which might be better than parsing of `PKG-INFO` (`pkginfo` package exists though, so it would be easy)

Either way, would like to hear the maintainers thoughts on that. If it something that should be fixed - I would try.

Guia de contribuição

Nenhum guia de contribuição indexado para este repositório

Avaliação

Esta issue ainda não foi avaliada.

Receba novas issues na sua caixa de entrada

Um resumo curto de issues do GitHub para quem está começando.