clearlydefined / clearlydefined/crawler

Still seeing some partial harvests

Open
#436 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
JavaScript
Stars
60
Forks
38
Avg merge
1h 49m
Merged PRs (30d)
1

Description

Although the "partial harvest" situation has improved dramatically since upgrading the memory of the containers used by the crawler, we are still seeing some components remain in the "partial" state after requeuing a harvest.

These include:

**git/github/prometheus/client_golang/8184d76b3b0bd3b01ed903690431ccb6826bf3e0 - PARTIAL**

Crawler logs report: ERROR - Scancode is reporting ENOMEM Error: spawn ENOMEM at ChildProcess.spawn

**git/github/SAP/vscode-webview-rpc-lib/90dc1f34fa9615667922ac3600bdd65cd07892f9 - PARTIAL**

Crawler logs report: ERROR - Scancode is reporting ENOMEM Error: spawn ENOMEM at ChildProcess.spawn

**git/github/googleapis/gnostic/896953e6749863beec38e27029c804e88c3144b8 - PARTIAL**

Crawler logs report: ERROR - Scancode is reporting ENOMEM Error: spawn ENOMEM at ChildProcess.spawn

Last time we upgraded the crawler infrastructure, we scaled UP by increasing the memory used by the containers the crawler runs in. We may need to scale OUT as well so that it is less likely for a crawler instance to be running so many harvests that it runs out of memory when a larger harvest occurs.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.