aboutcode-org / aboutcode-org/scancode-toolkit
Consolidating large codebases uses excessive amounts of memory
Open
bug
summaries
- Dominant language
- Python
- Stars
- 2.6k
- Forks
- 791
- Avg merge
- 1d 12h
- Merged PRs (30d)
- 5
Description
I was using the consolidation plugin on a 18.3 mb scan with 14886 files and scancode managed to use 10+ gigs of ram. I have a feeling that it is this dict (https://github.com/nexB/scancode-toolkit/blob/develop/src/summarycode/plugin_consolidate.py#L391) bloating up horribly when there are a lot of copyright holders consolidated.
Contributor guide
Assessment
This issue has not been assessed yet.