aboutcode-org / aboutcode-org/python-inspector

`packages` is empty when using a non-pypi.org simple index (e.g. Sonatype Nexus)

Abierto
#268 1 comentario 0 reacciones 0 asignados Ver en GitHub
Lenguaje dominante
Python
Estrellas
23
Forks
28
Métricas de merge de PR
Sin PR fusionados en 30 d

Descripción

Currently the metadata fetch requires the pypi.org-specific JSON API. (See also #260 )

When `PYINSP_INDEX_URL` points at a non-pypi.org simple index (Sonatype Nexus PyPI in my case), `python-inspector` resolves dependencies but the `packages` array in the output is empty - package metadata is never fetched.

For example: `PYINSP_INDEX_URL=https://my-nexus-pypi.com/repository/pypi/simple`:

```shell
root@2ae255f06ad8:/project# python-inspector -p3.13 -rrequirements.txt -olinux --json-pdt /tmp/test.json --verbose
...
retrieve package data from pypi:
retrieved package 'pkg:pypi/cx-oracle@7.3.0'
retrieved package 'pkg:pypi/redis@2.10.6'
done!

root@2ae255f06ad8:/project# jq '.packages[].name' /tmp/test.json
# (no output - packages is [])
```

Note the verbose log claims `retrieved package ...`, yet `packages` is empty - the retrieval silently failed.

Without the index (default pypi.org) it works:

```shell
root@2ae255f06ad8:/project# env -u PYINSP_INDEX_URL python-inspector -p3.13 -rrequirements.txt -olinux --json-pdt /tmp/test.json --verbose
...
root@2ae255f06ad8:/project# jq '.packages[].name' /tmp/test.json
"cx-oracle"
"redis"
```

Root cause is the `get_pypi_data_from_purl` builds the metadata URL from the index base and queries the pypi.org-specific JSON API, then returns `None` on any non-200:

https://github.com/aboutcode-org/python-inspector/blob/a841c7ce82c6087695af7fc558f5ad30fb5b1e27/src/python_inspector/package_data.py#L65-L77

With a Nexus index this resolves to `https://my-nexus-pypi.com/repository/pypi/pypi/redis/2.10.6/json`, which 404s - Nexus's PyPI format implements the simple index (PEP 503), not the `/pypi///json` API - so the function bails out and no `PackageData` is produced.

I initially assumed `--use-pypi-json-api` was the toggle for this. It doesn't help here: in its default (off) position `get_pypi_data_from_purl` still unconditionally queries the JSON API, and in the on position the help states `--index-url` is ignored (i.e. pypi.org), so it can't reach a private index either way.

I'm not sure what the correct approach would be, I feel like `--use-pypi-json-api` option should be fixed that way we download the selected sdist/wheel and parse its `PKG-INFO` for `License` / `Home-page` / `Author` / `Requires-Dist`. The simple index already provides the file list + digests, so the JSON API's `urls` are largely redundant; its unique value is the rich `info` block, which `PKG-INFO` also carries.

While searching for solution I've also stumbled upon `PEP 691` https://packaging.python.org/en/latest/specifications/simple-repository-api/#json-based-simple-api-for-python-package-indexes
Which might be better than parsing of `PKG-INFO` (`pkginfo` package exists though, so it would be easy)

Either way, would like to hear the maintainers thoughts on that. If it something that should be fixed - I would try.

Guía de contribución

No hay ninguna guía de contribución indexada para este repositorio

Evaluación

Este issue todavía no se ha evaluado.

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.