aboutcode-org / aboutcode-org/scancode-toolkit

Consolidating large codebases uses excessive amounts of memory

Open
#1,894 0 comments 0 reactions 1 assignee Claimed by @JonoYang View on GitHub
bug summaries
Dominant language
Python
Stars
2.6k
Forks
791
Avg merge
1d 12h
Merged PRs (30d)
5

Description

I was using the consolidation plugin on a 18.3 mb scan with 14886 files and scancode managed to use 10+ gigs of ram. I have a feeling that it is this dict (https://github.com/nexB/scancode-toolkit/blob/develop/src/summarycode/plugin_consolidate.py#L391) bloating up horribly when there are a lot of copyright holders consolidated.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.