aboutcode-org / aboutcode-org/purldb

Performance issues when using `api/resources/filter_by_checksums`

Aberta
#551 1 comentário 0 reações 1 responsável Reivindicada por @JonoYang Ver no GitHub
Linguagem predominante
HTML
Estrelas
67
Forks
69
Merge médio
8d 8h
PRs com merge (30d)
1

Descrição

On certain large purldb instances, when using the `api/resources/filter_by_checksums` endpoint via the scancode.io `map_deploy_to_develop` pipeline, the `match_to_purldb_resource` step is very slow and can take +30 hours to complete.

After debugging, we found that the two biggest reasons for the slowness are:

- Ordering of Resources, a lot of CPU time is spent ordering resources from a query
- Decoding large JSON fields, a lot of time is spent parsing JSON fields if they are too big, like the history field on Package

Immediate solutions that come to mind:

- Remove ordering for Resources
- Create proper History model for Package, expedient thing would be to empty history json field. Look into using `.only()` on queries.

Guia de contribuição

Nenhum guia de contribuição indexado para este repositório

Avaliação

Esta issue ainda não foi avaliada.

Receba novas issues na sua caixa de entrada

Um resumo curto de issues do GitHub para quem está começando.