clearlydefined / clearlydefined/crawler

ScanCode tool not triggered for a coordinate - aircompressor

Open
#798 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
JavaScript
Stars
60
Forks
38
Avg merge
1h 49m
Merged PRs (30d)
1

Description

The ScanCode tool is failing to trigger for the following coordinate:

git/github/airlift/aircompressor/e139a31be059855a5a70f6994a4ecaa0adbbea7a

**Scancode Investigation Summary**

**Root Cause**
- Large binary test files (`mozilla`, `samba`, `webster` ~50-100MB each in `testdata/silesia/`)
- 2 parallel scancode workers each loading files into memory
- 1000s timeout allows full processing of these huge files

### All Test Scenarios

| # | Scenario | Timeout | Processes | Options | Docker RAM | Peak Memory | Duration | Files Completed | Result |
|---|----------|---------|-----------|---------|------------|-------------|----------|-----------------|--------|
| 1 | **Original failure** | 1000s | 2 | - | 7.65GB | >7.65GB | N/A | 0 | OOM SIGKILL |
| 2 | Manual docker exec | 1000s | 0 | - | 47GB | ~17GB | ~22 min | 0/3 (all timed out) | Success |
| 3 | Test: huge timeout | 1000000s | 1 | `--quiet` | 16GB | >16GB | N/A | N/A | OOM SIGKILL |
| 4 | Test: huge timeout | 1000000s | 1 | `--quiet` | 47GB | ~36GB+ | N/A | N/A | OOM SIGKILL |
| 5 | Test: `--ignore` patterns | 1000000s | 2 | `--ignore */testdata/*` etc. | 16GB | ~2GB | ~2 min | N/A (skipped) | Success (files excluded) |
| 6 | Test: default timeout | 120s | 2 | - | 47GB | ~2GB | ~5 min | 0/3 (all timed out) | Success |
| 7 | **Final: original config** | 1000s | 2 | - | 47GB | **27.57GB** | ~33 min | 2/3 (samba timed out) | Success |

**Key Findings**

- Original config (`timeout: 1000`, `processes: 2`) needs **~28GB RAM**
- Original Docker limit was **7.65GB** -> OOM
- Manual run with `processes: 0` (debug mode) uses less memory (~17GB) but slower
- Default timeout (120s) skips large test files, only needs ~2GB
- `--ignore` patterns can exclude problematic files entirely (~2GB, fastest)
- Huge timeout (1000000s) forces full binary parsing → 36GB+ → OOM even at 47GB

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.