aboutcode-org / aboutcode-org/scancode-toolkit

Improve PyPI package declared license detection

Open
#2,487 0 comments 0 reactions 0 assignees View on GitHub
improve-license-detection license scan new feature package scan package-formats
Dominant language
Python
Stars
2.6k
Forks
791
Avg merge
1d 12h
Merged PRs (30d)
5

Description

The goal of this ticket is to improve PyPI package license detection across the board. While scancode-toolkit's PyPI package detection is pretty good, there a few repeat cases where license information is not properly gathered from PyPI package metadata. Usually this is because a `declared_license` value contains things we did not expect (like a URL) or is improperly formed.

Resolving this would likely require a mix of:
- adding new license detection rules to scancode,
- adding new and improved code to handle the specific patterns of license,
- creating new license mappings
- and possibly working with upstream maintainers to improve their license declarations.

The approach should be to start with a complete data set of all package manifests and find patterns of license issues and establish the baseline, possibly with classifiers and ML. The end results should be a significant improvement to the license detection quality for the PyPI packages.

This https://github.com/pypa/bandersnatch/ and the PyPI API may help collect a list of all declared licenses.

See also https://www.python.org/dev/peps/pep-0639/ and may be https://github.com/nexB/scancode-toolkit/issues/253 too

There are also other related ticket for other package types such as:
- https://github.com/nexB/scancode-toolkit/issues/2412 for RPM that has some detailed examples

And a project idea: https://github.com/nexB/aboutcode/wiki/Project-Ideas-Improve-PyPI-package-license-detection

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.