aboutcode-org / aboutcode-org/purldb

Performance issues when using `api/resources/filter_by_checksums`

未關閉
#551 1 則留言 0 個 reaction 已指派 1 人 已被 @JonoYang 認領 在 GitHub 檢視
主要語言
HTML
星號
67
分支
69
平均合併
8 天 8 小時
30 天內合併 PR
1

描述

On certain large purldb instances, when using the `api/resources/filter_by_checksums` endpoint via the scancode.io `map_deploy_to_develop` pipeline, the `match_to_purldb_resource` step is very slow and can take +30 hours to complete.

After debugging, we found that the two biggest reasons for the slowness are:

- Ordering of Resources, a lot of CPU time is spent ordering resources from a query
- Decoding large JSON fields, a lot of time is spent parsing JSON fields if they are too big, like the history field on Package

Immediate solutions that come to mind:

- Remove ordering for Resources
- Create proper History model for Package, expedient thing would be to empty history json field. Look into using `.only()` on queries.

貢獻指南

這個儲存庫沒有索引到貢獻指南

評估

這個 Issue 還沒有評估資料。

把新 issue 寄到你的電子郵件信箱

精選適合新手參與的 GitHub issue 摘要。