aboutcode-org / aboutcode-org/scancode-toolkit

Improve false positive license detection for license lists

Open
#2,651 12 comments 0 reactions 0 assignees View on GitHub
improve-license-detection license scan new feature
Dominant language
Python
Stars
2.6k
Forks
791
Avg merge
1d 12h
Merged PRs (30d)
5

Description

We have lists of license identifiers in code or data files that are being detected and lead to many false positive (FP). These are typically list of SPDX identifiers and are mostly found in license-related tools... or package management tools. But these tools are seen everywhere.

See this attached example code from NuGet: [NuGetLicenseData.cs.txt](https://github.com/nexB/scancode-toolkit/files/7019923/NuGetLicenseData.cs.txt) with long lists like this:
```C#
{ "AGPL-1.0", new LicenseData(licenseID: "AGPL-1.0", isOsiApproved: false, isDeprecatedLicenseId: true, isFsfLibre: true) },
{ "AGPL-1.0-only", new LicenseData(licenseID: "AGPL-1.0-only", isOsiApproved: false, isDeprecatedLicenseId: false, isFsfLibre: false) },
{ "AGPL-1.0-or-later", new LicenseData(licenseID: "AGPL-1.0-or-later", isOsiApproved: false, isDeprecatedLicenseId: false, isFsfLibre: false) },
```

For now, adding new "false positive" license detection rules has been the solution to deal with list of license keys such as
- #1032
- #797
- #2647
- #2515

The problem is a bit related to
- #2403
... in the sense that the context of where we find a licenses matters: for bare words, the case, being surrounded by gibberish or found in a binary may be a false positive clue.

For instance the fact that we are in code literals like in #2502 could be a clue that this an FP

We should find a better way than just adding many new false positive rules like in 5f39252c3c817cb26bfbf7f93ceea08d8ab0c3a2 and #2505 ... some ideas:
- Heuristics when several licenses are detected in alphabetical order using most license names
- ML ?
- list of known packages Purl that have these issues (think about what would happen if you scan ScanCode :roll_eyes: )

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.