aboutcode-org / aboutcode-org/scancode.io

D2D: Improve the "map_checksum" step

Ouverte
#907 0 commentaires 0 réactions 0 personnes assignées Voir sur GitHub
Langage dominant
Python
Étoiles
215
Forks
203
Merge moyen
4 j 8 h
PR mergées (30 j)
6

Description

The "map_checksum" step first creates a query on the To/ and then match each To/ Resource to the whole From/ Resources. If there are 500,000 To/ Resources we would be doing 500K queries. This can takes several hours on large codebases and the 500K query is the obvious culprit.

This should be optimizable by only issuing a single query for matching that would join the To/ and From/ on checksum and return the results grouped by To/.
From there on, the creation of relations could be batched for further speed up.

Guide de contribution

Ouvrir le guide de contribution

Évaluation

Cette issue n'a pas encore été évaluée.

Recevez les nouvelles issues par e-mail

Un résumé court des issues GitHub adaptées aux débutants.