aboutcode-org / aboutcode-org/scancode-toolkit

Package scan for open source component detects 940mb of duplicated information

Open
#2,915 2 comments 0 reactions 0 assignees View on GitHub
bug package scan
Dominant language
Python
Stars
2.6k
Forks
791
Avg merge
1d 12h
Merged PRs (30d)
5

Description

### Description

We encountered an issue trying to scan the following open source component: [RxDataSources - 4.0.1](https://github.com/RxSwiftCommunity/RxDataSources/tree/4.0.1)

When doing a scan with the package information enabled it detects about 940MB of package information in the file [RxDataSources.podspec](https://github.com/RxSwiftCommunity/RxDataSources/blob/4.0.1/RxDataSources.podspec).

The output (the value of the package field in the resulting json file for this specific file) looks like this, Scancode seems to catch on the same line multiple times:
![image](https://user-images.githubusercontent.com/10448053/162915364-07e71b8a-d992-49ac-a3e8-bc14b3939339.png)

### How To Reproduce

Download sourcecode from [RxDataSources - 4.0.1](https://github.com/RxSwiftCommunity/RxDataSources/tree/4.0.1)
Run Scancode on the sourcecode with package information enabled

The exact command we are using:
scancode -clpi --license-score 65 --max-in-memory -1 -n 4 --strip-root --verbose --json-pp ./out.json path/to/sourcecode

### System configuration

* What OS are you running on? Tested on both Windows and dockerized Linux
* What version of scancode-toolkit was used to generate the scan file? [scancode-toolkit-30.1.0_py36-windows.zip](https://github.com/nexB/scancode-toolkit/releases/download/v30.1.0/scancode-toolkit-30.1.0_py36-windows.zip) for Windows and [scancode-toolkit-30.1.0_py36-linux.tar.xz](https://github.com/nexB/scancode-toolkit/releases/download/v30.1.0/scancode-toolkit-30.1.0_py36-linux.tar.xz) for Linux.
* What installation method was used to install/run scancode? For windows: Source download. For Linux: Source download -> Docker build

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.