aboutcode-org / aboutcode-org/scancode.io

package assembly in `inspect_packages` and other pipelines is too slow

Aberta
#1,398 7 comentários 0 reações 1 responsável Reivindicada por @AyanSinhaMahapatra Ver no GitHub
bug performance
Linguagem predominante
Python
Estrelas
215
Forks
203
Merge médio
4d 8h
PRs com merge (30d)
6

Descrição

I am running the `inspect_packages` pipeline on a very large codebase that is an npm package. The pipeline takes a very long time at the `scan_for_application_packages` step on the package assembly portion of package scanning (https://github.com/aboutcode-org/scancode.io/blob/main/scanpipe/pipes/scancode.py#L459), where we are running this code (https://github.com/aboutcode-org/scancode-toolkit/blob/develop/src/packagedcode/npm.py#L82)

Since this codebase is an npm package, we pretty much consider all files in it to be part of the npm package. We are running code that is originally intended for a scancode-toolkit codebase. Walking a codebase using sctk code on a scio codebase is not performant because each call to a sctk codebase traversal method is an individual query to the database. The methods here are called multiple times and `.save()` is called on each Resource when performing package assembly.

Guia de contribuição

Abrir o guia de contribuição

Avaliação

Esta issue ainda não foi avaliada.

Receba novas issues na sua caixa de entrada

Um resumo curto de issues do GitHub para quem está começando.