aboutcode-org / aboutcode-org/scancode.io

D2D: Improve the "map_checksum" step

Open
#907 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
215
Forks
203
Avg merge
4d 8h
Merged PRs (30d)
6

Description

The "map_checksum" step first creates a query on the To/ and then match each To/ Resource to the whole From/ Resources. If there are 500,000 To/ Resources we would be doing 500K queries. This can takes several hours on large codebases and the 500K query is the obvious culprit.

This should be optimizable by only issuing a single query for matching that would join the To/ and From/ on checksum and return the results grouped by To/.
From there on, the creation of relations could be batched for further speed up.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.