aboutcode-org / aboutcode-org/scancode-toolkit

Improve false positive license detection for license lists

Aberta
#2,651 12 comentários 0 reações 0 responsáveis Ver no GitHub
improve-license-detection license scan new feature
Linguagem predominante
Python
Estrelas
2.6k
Forks
791
Merge médio
1d 12h
PRs com merge (30d)
5

Descrição

We have lists of license identifiers in code or data files that are being detected and lead to many false positive (FP). These are typically list of SPDX identifiers and are mostly found in license-related tools... or package management tools. But these tools are seen everywhere.

See this attached example code from NuGet: [NuGetLicenseData.cs.txt](https://github.com/nexB/scancode-toolkit/files/7019923/NuGetLicenseData.cs.txt) with long lists like this:
```C#
{ "AGPL-1.0", new LicenseData(licenseID: "AGPL-1.0", isOsiApproved: false, isDeprecatedLicenseId: true, isFsfLibre: true) },
{ "AGPL-1.0-only", new LicenseData(licenseID: "AGPL-1.0-only", isOsiApproved: false, isDeprecatedLicenseId: false, isFsfLibre: false) },
{ "AGPL-1.0-or-later", new LicenseData(licenseID: "AGPL-1.0-or-later", isOsiApproved: false, isDeprecatedLicenseId: false, isFsfLibre: false) },
```

For now, adding new "false positive" license detection rules has been the solution to deal with list of license keys such as
- #1032
- #797
- #2647
- #2515

The problem is a bit related to
- #2403
... in the sense that the context of where we find a licenses matters: for bare words, the case, being surrounded by gibberish or found in a binary may be a false positive clue.

For instance the fact that we are in code literals like in #2502 could be a clue that this an FP

We should find a better way than just adding many new false positive rules like in 5f39252c3c817cb26bfbf7f93ceea08d8ab0c3a2 and #2505 ... some ideas:
- Heuristics when several licenses are detected in alphabetical order using most license names
- ML ?
- list of known packages Purl that have these issues (think about what would happen if you scan ScanCode :roll_eyes: )

Guia de contribuição

Abrir o guia de contribuição

Avaliação

Esta issue ainda não foi avaliada.

Receba novas issues na sua caixa de entrada

Um resumo curto de issues do GitHub para quem está começando.