aboutcode-org / aboutcode-org/scancode.io

package assembly in `inspect_packages` and other pipelines is too slow

Đang mở
#1,398 7 bình luận 0 reaction 1 người được giao Được @AyanSinhaMahapatra nhận Xem trên GitHub
bug performance
Ngôn ngữ chính
Python
Star
215
Fork
203
Merge trung bình
4 ngày 8 giờ
Pull request đã merge (30 ngày)
6

Mô tả

I am running the `inspect_packages` pipeline on a very large codebase that is an npm package. The pipeline takes a very long time at the `scan_for_application_packages` step on the package assembly portion of package scanning (https://github.com/aboutcode-org/scancode.io/blob/main/scanpipe/pipes/scancode.py#L459), where we are running this code (https://github.com/aboutcode-org/scancode-toolkit/blob/develop/src/packagedcode/npm.py#L82)

Since this codebase is an npm package, we pretty much consider all files in it to be part of the npm package. We are running code that is originally intended for a scancode-toolkit codebase. Walking a codebase using sctk code on a scio codebase is not performant because each call to a sctk codebase traversal method is an individual query to the database. The methods here are called multiple times and `.save()` is called on each Resource when performing package assembly.

Hướng dẫn đóng góp

Mở hướng dẫn đóng góp

Đánh giá

Issue này chưa được đánh giá.

Nhận issue mới trong hộp thư của bạn

Bản tóm tắt ngắn những issue GitHub phù hợp với người mới.