aboutcode-org / aboutcode-org/scancode-toolkit

Improve false positive license detection for license lists

Offen
#2,651 12 Kommentare 0 Reaktionen 0 zugewiesene Personen Auf GitHub ansehen
improve-license-detection license scan new feature
Vorherrschende Sprache
Python
Sterne
2.6k
Forks
791
Ø Merge
1 T. 12 Std.
Gemergte PRs (30 T.)
5

Beschreibung

We have lists of license identifiers in code or data files that are being detected and lead to many false positive (FP). These are typically list of SPDX identifiers and are mostly found in license-related tools... or package management tools. But these tools are seen everywhere.

See this attached example code from NuGet: [NuGetLicenseData.cs.txt](https://github.com/nexB/scancode-toolkit/files/7019923/NuGetLicenseData.cs.txt) with long lists like this:
```C#
{ "AGPL-1.0", new LicenseData(licenseID: "AGPL-1.0", isOsiApproved: false, isDeprecatedLicenseId: true, isFsfLibre: true) },
{ "AGPL-1.0-only", new LicenseData(licenseID: "AGPL-1.0-only", isOsiApproved: false, isDeprecatedLicenseId: false, isFsfLibre: false) },
{ "AGPL-1.0-or-later", new LicenseData(licenseID: "AGPL-1.0-or-later", isOsiApproved: false, isDeprecatedLicenseId: false, isFsfLibre: false) },
```

For now, adding new "false positive" license detection rules has been the solution to deal with list of license keys such as
- #1032
- #797
- #2647
- #2515

The problem is a bit related to
- #2403
... in the sense that the context of where we find a licenses matters: for bare words, the case, being surrounded by gibberish or found in a binary may be a false positive clue.

For instance the fact that we are in code literals like in #2502 could be a clue that this an FP

We should find a better way than just adding many new false positive rules like in 5f39252c3c817cb26bfbf7f93ceea08d8ab0c3a2 and #2505 ... some ideas:
- Heuristics when several licenses are detected in alphabetical order using most license names
- ML ?
- list of known packages Purl that have these issues (think about what would happen if you scan ScanCode :roll_eyes: )

Beitragsleitfaden

Beitragsleitfaden öffnen

Bewertung

Dieses Issue wurde noch nicht bewertet.

Neue Issues direkt in Ihr Postfach

Eine kurze Übersicht über anfängerfreundliche GitHub-Issues.