clearlydefined / clearlydefined/crawler
Still seeing some partial harvests
- Dominant language
- JavaScript
- Stars
- 60
- Forks
- 38
- Avg merge
- 1h 49m
- Merged PRs (30d)
- 1
Description
Although the "partial harvest" situation has improved dramatically since upgrading the memory of the containers used by the crawler, we are still seeing some components remain in the "partial" state after requeuing a harvest.
These include:
**git/github/prometheus/client_golang/8184d76b3b0bd3b01ed903690431ccb6826bf3e0 - PARTIAL**
Crawler logs report: ERROR - Scancode is reporting ENOMEM Error: spawn ENOMEM at ChildProcess.spawn
**git/github/SAP/vscode-webview-rpc-lib/90dc1f34fa9615667922ac3600bdd65cd07892f9 - PARTIAL**
Crawler logs report: ERROR - Scancode is reporting ENOMEM Error: spawn ENOMEM at ChildProcess.spawn
**git/github/googleapis/gnostic/896953e6749863beec38e27029c804e88c3144b8 - PARTIAL**
Crawler logs report: ERROR - Scancode is reporting ENOMEM Error: spawn ENOMEM at ChildProcess.spawn
Last time we upgraded the crawler infrastructure, we scaled UP by increasing the memory used by the containers the crawler runs in. We may need to scale OUT as well so that it is less likely for a crawler instance to be running so many harvests that it runs out of memory when a larger harvest occurs.
Contributor guide
Assessment
This issue has not been assessed yet.