aboutcode-org / aboutcode-org/scancode-toolkit

ScanCode process not terminating after having finished

Offen
#4,134 3 Kommentare 1 Reaktion 0 zugewiesene Personen Auf GitHub ansehen
bug
Vorherrschende Sprache
Python
Sterne
2.6k
Forks
791
Ø Merge
1 T. 12 Std.
Gemergte PRs (30 T.)
5

Beschreibung

### Description

We are scanning a large git repository that also has 3 submodules.
We execute the scan within a OpenShift pod, using

Python 3.11
ScanCode 32.3.1

The ScanCode scan runs as part of an ORT (OSS Review Toolkit) scan.
The following command line is used:

_scancode --copyright --license --info --strip-root --timeout 300 --max-in-memory 5000 --processes 4 --license-references --json /tmp/ort-ScanCode8749968816967782004/result.json /tmp/ort-DefaultProvenanceDownloader17726939893427938442_

As processes on the OpenShift pod, we see
* A java process (ORT worker)
* One Python process (the ScanCode CLI)
* 4 Python child processes

For some time, we see all the processes busy doing their work (using the tool _top_) , but at some point in time, the processes stop to work and are displayed in status _sleep_. This means, the scan job just hangs until eternity.

But on console output, we see actually that ScanCode has finished. It even cleaned up files:

````
$ find . -name scancode.stderr -exec cat {} \;
Setup plugins...
Collect file inventory...
Scan files for: info, licenses, copyrights with 4 process(es)...
Scanning done.
Summary: info, licenses, copyrights with 4 process(es)
Errors count: 0
Scan Speed: 7.35 files/sec. 104.36 KB/sec.
Initial counts: 15172 resource(s): 11782 file(s) and 3390 directorie(s)
Final counts: 15171 resource(s): 11782 file(s) and 3389 directorie(s) for 163.43 MB
Timings:
scan_start: 2025-02-06T092555.935638
scan_end: 2025-02-06T095310.228719
setup_scan:licenses: 4.85s
setup: 4.86s
inventory: 8.59s
scan:licenses: 5.89s
scan: 1603.61s
post-scan:license-references: 1.19s
post-scan: 1.19s
output:json: 4.94s
output: 4.94s
total: 1640.31s
Removing temporary files...done.
Setup plugins...
Collect file inventory...
Scan files for: info, licenses, copyrights with 4 process(es)...
````

So, for me (not being proficient in Python) it looks like the mother python process is finished and does a _ctx.exit(rc)_, but the process actually does not terminate. I guess it does not terminate, because the child processes are still alive.

Because the ScanCode mother process does not terminate, also the ORT scan step hangs.

### How To Reproduce

Sorry, the problem is reproduceable with an internal repository of ours, but I can give no simple way how to reproduce.

Nevertheless I decided to write this bug ticket, because others may run into the same situation.

Beitragsleitfaden

Beitragsleitfaden öffnen

Bewertung

Dieses Issue wurde noch nicht bewertet.

Neue Issues direkt in Ihr Postfach

Eine kurze Übersicht über anfängerfreundliche GitHub-Issues.