aboutcode-org / aboutcode-org/scancode-toolkit

Inconsistent URL detection

Open
#4,243 2 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Python
Stars
2.6k
Forks
791
Avg merge
1d 12h
Merged PRs (30d)
5

Description

### Description

Scanning for URLs does not seem to be consistent and I sometimes see expected and sometimes unexpected results.

Let's take the following files as an example:

* https://github.com/RDFLib/rdflib/blob/9f76b0fc7b9d6a01cc2e08cad29ac54acb60c026/rdflib/plugins/serializers/trig.py does not report http://www.w3.org/TR/trig/
* https://github.com/RDFLib/rdflib/blob/9f76b0fc7b9d6a01cc2e08cad29ac54acb60c026/rdflib/plugins/serializers/trix.py reports http://www.w3.org/2004/03/trix/trix-1/, but not http://www.w3.org/XML/1998/namespace
* https://github.com/RDFLib/rdflib/blob/9f76b0fc7b9d6a01cc2e08cad29ac54acb60c026/rdflib/plugins/serializers/turtle.py reports http://www.w3.org/TeamSubmission/turtle/ correctly.

### How To Reproduce

Run `scancode.api.get_urls(path)` on the corresponding files.

### System configuration

> For bug reports, it really helps us to know:

* What OS are you running on? Linux
* What version of scancode-toolkit was used to generate the scan file? 32.3.3
* What installation method was used to install/run scancode? pip

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.