aboutcode-org / aboutcode-org/scancode.io
D2D: Improve the "map_checksum" step
Đang mở
- Ngôn ngữ chính
- Python
- Star
- 215
- Fork
- 203
- Merge trung bình
- 4 ngày 8 giờ
- Pull request đã merge (30 ngày)
- 6
Mô tả
The "map_checksum" step first creates a query on the To/ and then match each To/ Resource to the whole From/ Resources. If there are 500,000 To/ Resources we would be doing 500K queries. This can takes several hours on large codebases and the 500K query is the obvious culprit.
This should be optimizable by only issuing a single query for matching that would join the To/ and From/ on checksum and return the results grouped by To/.
From there on, the creation of relations could be batched for further speed up.
Hướng dẫn đóng góp
Đánh giá
Issue này chưa được đánh giá.