aboutcode-org / aboutcode-org/scancode-toolkit

Performance degradation and memory usage in 32.5.0_py3.14

Aperta
#4,709 8 commenti 3 reazioni 0 assegnatari Vedi su GitHub
bug
Lingua principale
Python
Stelle
2.6k
Fork
791
Merge medio
1g 12h
PR unite (30g)
5

Descrizione

### Summary
Performance and memory usage issues in the Python 3.14 release compared to the 3.13 one:
- Scanning speed is down about 40 %
- Memory usage for a scan using 21 processes is increased from 3.1 GB to 25 GB.

### Description

We have discovered performance and memory usage issues in the Python 3.14 release of v32.5.0 that prevents us from upgrading to that version. Most of these issues are not present in the Python 3.13 release or in v32.4.1.

The way we run Scancode, in a containerized environment, seems to exacerbate the problem. A scan that took 30 seconds in 32.4.1 now takes 4 minutes. Passive memory usage is about 50% higher, and memory usage during maximum load is around 250% higher. Some of these issues may be attributed to our environment, but we're seeing similar problems when running Scancode in a non-containerized fashion, using the CLI.

Running Scancode directly on my laptop gives the following results when scanning the [async 3.2.6 npm package](https://www.npmjs.com/package/async):

#### ScanCode v32.5.0_py3.14
```
$ ./scancode -l --json output.json ./package
Setup plugins...
Collect file inventory...
Scan files for: licenses with 21 process(es)...
[####################] 274
Scanning done.
Summary: licenses with 21 process(es)
Errors count: 0
Scan Speed: 14.03 files/sec.
Initial counts: 140 resource(s): 137 file(s) and 3 directorie(s)
Final counts: 140 resource(s): 137 file(s) and 3 directorie(s)
Timings:
scan_start: 2026-01-28T131349.526492
scan_end: 2026-01-28T131400.641108
setup_scan:licenses: 1.33s
setup: 1.33s
scan: 9.76s
total: 11.12s
Removing temporary files...done.
```
Memory usage during the scan peaks at **25 GB (!)** - it seems to scale linearly with the number of processes.

#### ScanCode v32.5.0_py3.13
```
$ ./scancode -l --json output.json ./package
Setup plugins...
Collect file inventory...
Scan files for: licenses with 21 process(es)...
[####################] 274
Scanning done.
Summary: licenses with 21 process(es)
Errors count: 0
Scan Speed: 23.23 files/sec.
Initial counts: 140 resource(s): 137 file(s) and 3 directorie(s)
Final counts: 140 resource(s): 137 file(s) and 3 directorie(s)
Timings:
scan_start: 2026-01-28T131911.340798
scan_end: 2026-01-28T131918.676450
setup_scan:licenses: 1.42s
setup: 1.42s
scan: 5.90s
total: 7.37s
Removing temporary files...done.
```
Memory usage peaks at 3.1 GB. While it it less when only running with one process, the difference is nowhere near the one in v32.5.0_py3.14.

#### ScanCode v32.4.1_py3.13
```
$ ./scancode -l --json output.json ./package
Setup plugins...
Collect file inventory...
Scan files for: licenses with 21 process(es)...
[####################] 274
Scanning done.
Summary: licenses with 21 process(es)
Errors count: 0
Scan Speed: 23.04 files/sec.
Initial counts: 140 resource(s): 137 file(s) and 3 directorie(s)
Final counts: 140 resource(s): 137 file(s) and 3 directorie(s)
Timings:
scan_start: 2026-01-28T135229.947719
scan_end: 2026-01-28T135237.361817
setup_scan:licenses: 1.45s
setup: 1.45s
scan: 5.95s
total: 7.45s
Removing temporary files...done.
```
Memory usage peaks at 3.1 GB, same as with v32.5.0_py3.13.

Copyright scans are a bit slower as well, but they don't seem to have the same memory issues.

### How To Reproduce

```
curl https://registry.npmjs.org/async/-/async-3.2.6.tgz | tar -xz
scancode -l --json output.json ./package
```

### System configuration

- Ubuntu 24.04
- Scancode downloaded from the release page (https://github.com/aboutcode-org/scancode-toolkit/releases/download/v32.5.0/scancode-toolkit-v32.5.0_py3.14-linux.tar.gz)

Guida per i contributori

Apri la guida per i contributori

Valutazione

Questa issue non è ancora stata valutata.

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.