aboutcode-org / aboutcode-org/scancode.io

PurlDB-requested package index scans can fail for larger packages with large scans and many scans at once

未關閉
#1,143 0 則留言 0 個 reaction 已指派 0 人 在 GitHub 檢視
主要語言
Python
星號
215
分支
203
平均合併
4 天 8 小時
30 天內合併 PR
6

描述

`send_scan_project_results` in `purldb-scan-queue-worker` isn't able to properly send the large scan results.

This issue fixed the critical problems:
- https://github.com/nexB/purldb/issues/362

And we can still improve this, using some of these approaches:

- [ ] Send compressed scan results: I was able to compress the ~350MB scan result to under 25MB using zstd or gzip ... xz is way too slow otherwise. For a start using the built in Gzip compression of HTTP clients and servers should help a lot.
- [ ] Adapt timeout to payload: The 60-second `DEFAULT_TIMEOUT` is not sufficient for sending scan results, we should have a variable timeout depending on the size of the scan results.
- [ ] Consider paginated upload in multiple chunks or streamed upload using JSON lines.

貢獻指南

開啟貢獻指南

評估

這個 Issue 還沒有評估資料。

把新 issue 寄到你的電子郵件信箱

精選適合新手參與的 GitHub issue 摘要。