aboutcode-org / aboutcode-org/scancode-toolkit

ScanCode process not terminating after having finished

Open
#4,134 3 comments 1 reaction 0 assignees View on GitHub
bug
Dominant language
Python
Stars
2.6k
Forks
791
Avg merge
1d 12h
Merged PRs (30d)
5

Description

### Description

We are scanning a large git repository that also has 3 submodules.
We execute the scan within a OpenShift pod, using

Python 3.11
ScanCode 32.3.1

The ScanCode scan runs as part of an ORT (OSS Review Toolkit) scan.
The following command line is used:

_scancode --copyright --license --info --strip-root --timeout 300 --max-in-memory 5000 --processes 4 --license-references --json /tmp/ort-ScanCode8749968816967782004/result.json /tmp/ort-DefaultProvenanceDownloader17726939893427938442_

As processes on the OpenShift pod, we see
* A java process (ORT worker)
* One Python process (the ScanCode CLI)
* 4 Python child processes

For some time, we see all the processes busy doing their work (using the tool _top_) , but at some point in time, the processes stop to work and are displayed in status _sleep_. This means, the scan job just hangs until eternity.

But on console output, we see actually that ScanCode has finished. It even cleaned up files:

````
$ find . -name scancode.stderr -exec cat {} \;
Setup plugins...
Collect file inventory...
Scan files for: info, licenses, copyrights with 4 process(es)...
Scanning done.
Summary: info, licenses, copyrights with 4 process(es)
Errors count: 0
Scan Speed: 7.35 files/sec. 104.36 KB/sec.
Initial counts: 15172 resource(s): 11782 file(s) and 3390 directorie(s)
Final counts: 15171 resource(s): 11782 file(s) and 3389 directorie(s) for 163.43 MB
Timings:
scan_start: 2025-02-06T092555.935638
scan_end: 2025-02-06T095310.228719
setup_scan:licenses: 4.85s
setup: 4.86s
inventory: 8.59s
scan:licenses: 5.89s
scan: 1603.61s
post-scan:license-references: 1.19s
post-scan: 1.19s
output:json: 4.94s
output: 4.94s
total: 1640.31s
Removing temporary files...done.
Setup plugins...
Collect file inventory...
Scan files for: info, licenses, copyrights with 4 process(es)...
````

So, for me (not being proficient in Python) it looks like the mother python process is finished and does a _ctx.exit(rc)_, but the process actually does not terminate. I guess it does not terminate, because the child processes are still alive.

Because the ScanCode mother process does not terminate, also the ORT scan step hangs.

### How To Reproduce

Sorry, the problem is reproduceable with an internal repository of ours, but I can give no simple way how to reproduce.

Nevertheless I decided to write this bug ticket, because others may run into the same situation.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.