aboutcode-org / aboutcode-org/scancode-toolkit
Make creation of toplevel 'packages' field optional?
- Lingua principale
- Python
- Stelle
- 2.6k
- Fork
- 791
- Merge medio
- 1g 12h
- PR unite (30g)
- 5
Descrizione
TL;DR: Creation of the top-level `packages` information can be very costly. Are there downsides of skipping it (via CLI option), when it is not used?
## Context
We regularly scan huge codebases (upto O(500_000) files). In one of the more extreme cases, running Scancode takes ~22h. Inspecting the logs, I found that only ~2h were spent on actually scanning the files. The step that took the majority of runtime at ~20h was actually `Filter scan: packages....`. Some more logging revealed that almost all runtime is spent creating/populating the top-level `packages` key. Since we don't use this information at all, I simply disabled that code and the one below. Concretely I disabled the rest of `PackageScanner.process_codebase` [from this line on](https://github.com/aboutcode-org/scancode-toolkit/blob/4b57a7fe86e60e14694445a63f379b22ecbd7135/src/packagedcode/plugin_package.py#L271).
## Remaining questions:
1. While I did some tests that showed no significant differences in the `files` section, the code I disabled potentially does something to the `files` information by `add_referenced_license_detection_from_package` but I couldn't really understand what or in which cases. Is there a risk of significantly worsening results when disabling the toplevel `packages` section?
2. Is this problem relevant to more people? If so, I'll happily contribute a PR introducing a new CLI option to skip this code.
## Speculation about the origin of this problem
The problem simply might be that the project we are scanning has a lot of package information. The fact that this shows so dramatically here (2h scanning vs 20 hours post-processing package data) might be caused by the fact that we use 40 workers for scanning while the package code is purely single threaded. So theoretically, another possibility could be to not disable this code but instead use parallelization. With 40 workers this would ideally reduce the runtime from 20h to 30min. However, this piece of code seems to heavily mutate the `codebase` object so it might be rather difficult to employ parallelization.
suggested tags: performance, package-at-toplevel
Guida per i contributori
Apri la guida per i contributori
Valutazione
Questa issue non è ancora stata valutata.