aboutcode-org / aboutcode-org/scancode-toolkit

Make creation of toplevel 'packages' field optional?

オープン
#4,213 コメント 1 件 リアクション 1 件 担当者 0 名 GitHub で見る
主要言語
Python
スター
2.6k
フォーク
791
平均マージ
1日 12時間
マージ済み PR(30日)
5

説明

TL;DR: Creation of the top-level `packages` information can be very costly. Are there downsides of skipping it (via CLI option), when it is not used?

## Context
We regularly scan huge codebases (upto O(500_000) files). In one of the more extreme cases, running Scancode takes ~22h. Inspecting the logs, I found that only ~2h were spent on actually scanning the files. The step that took the majority of runtime at ~20h was actually `Filter scan: packages....`. Some more logging revealed that almost all runtime is spent creating/populating the top-level `packages` key. Since we don't use this information at all, I simply disabled that code and the one below. Concretely I disabled the rest of `PackageScanner.process_codebase` [from this line on](https://github.com/aboutcode-org/scancode-toolkit/blob/4b57a7fe86e60e14694445a63f379b22ecbd7135/src/packagedcode/plugin_package.py#L271).

## Remaining questions:
1. While I did some tests that showed no significant differences in the `files` section, the code I disabled potentially does something to the `files` information by `add_referenced_license_detection_from_package` but I couldn't really understand what or in which cases. Is there a risk of significantly worsening results when disabling the toplevel `packages` section?
2. Is this problem relevant to more people? If so, I'll happily contribute a PR introducing a new CLI option to skip this code.

## Speculation about the origin of this problem
The problem simply might be that the project we are scanning has a lot of package information. The fact that this shows so dramatically here (2h scanning vs 20 hours post-processing package data) might be caused by the fact that we use 40 workers for scanning while the package code is purely single threaded. So theoretically, another possibility could be to not disable this code but instead use parallelization. With 40 workers this would ideally reduce the runtime from 20h to 30min. However, this piece of code seems to heavily mutate the `codebase` object so it might be rather difficult to employ parallelization.

suggested tags: performance, package-at-toplevel

コントリビューションガイド

コントリビューションガイドを開く

評価

この issue はまだ評価されていません。

新しい issue をメールで受け取る

初心者向けの GitHub issue を短くまとめたダイジェスト。