aboutcode-org / aboutcode-org/scancode.io

D2D: Improve the "map_checksum" step

未關閉
#907 0 則留言 0 個 reaction 已指派 0 人 在 GitHub 檢視
主要語言
Python
星號
215
分支
203
平均合併
4 天 8 小時
30 天內合併 PR
6

描述

The "map_checksum" step first creates a query on the To/ and then match each To/ Resource to the whole From/ Resources. If there are 500,000 To/ Resources we would be doing 500K queries. This can takes several hours on large codebases and the 500K query is the obvious culprit.

This should be optimizable by only issuing a single query for matching that would join the To/ and From/ on checksum and return the results grouped by To/.
From there on, the creation of relations could be batched for further speed up.

貢獻指南

開啟貢獻指南

評估

這個 Issue 還沒有評估資料。

把新 issue 寄到你的電子郵件信箱

精選適合新手參與的 GitHub issue 摘要。