aboutcode-org / aboutcode-org/scancode.io

package assembly in `inspect_packages` and other pipelines is too slow

未關閉
#1,398 7 則留言 0 個 reaction 已指派 1 人 已被 @AyanSinhaMahapatra 認領 在 GitHub 檢視
bug performance
主要語言
Python
星號
215
分支
203
平均合併
4 天 8 小時
30 天內合併 PR
6

描述

I am running the `inspect_packages` pipeline on a very large codebase that is an npm package. The pipeline takes a very long time at the `scan_for_application_packages` step on the package assembly portion of package scanning (https://github.com/aboutcode-org/scancode.io/blob/main/scanpipe/pipes/scancode.py#L459), where we are running this code (https://github.com/aboutcode-org/scancode-toolkit/blob/develop/src/packagedcode/npm.py#L82)

Since this codebase is an npm package, we pretty much consider all files in it to be part of the npm package. We are running code that is originally intended for a scancode-toolkit codebase. Walking a codebase using sctk code on a scio codebase is not performant because each call to a sctk codebase traversal method is an individual query to the database. The methods here are called multiple times and `.save()` is called on each Resource when performing package assembly.

貢獻指南

開啟貢獻指南

評估

這個 Issue 還沒有評估資料。

把新 issue 寄到你的電子郵件信箱

精選適合新手參與的 GitHub issue 摘要。