aboutcode-org / aboutcode-org/scancode.io

PurlDB-requested package index scans can fail for larger packages with large scans and many scans at once

Open
#1,143 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
215
Forks
203
Avg merge
4d 8h
Merged PRs (30d)
6

Description

`send_scan_project_results` in `purldb-scan-queue-worker` isn't able to properly send the large scan results.

This issue fixed the critical problems:
- https://github.com/nexB/purldb/issues/362

And we can still improve this, using some of these approaches:

- [ ] Send compressed scan results: I was able to compress the ~350MB scan result to under 25MB using zstd or gzip ... xz is way too slow otherwise. For a start using the built in Gzip compression of HTTP clients and servers should help a lot.
- [ ] Adapt timeout to payload: The 60-second `DEFAULT_TIMEOUT` is not sufficient for sending scan results, we should have a variable timeout depending on the size of the scan results.
- [ ] Consider paginated upload in multiple chunks or streamed upload using JSON lines.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.