aboutcode-org / aboutcode-org/scancode-toolkit

Disk cache usage is slow

Ouverte
#1,014 3 commentaires 0 réactions 0 personnes assignées Voir sur GitHub
core and api documentation performance Priority: medium
Langage dominant
Python
Étoiles
2.6k
Forks
791
Merge moyen
1 j 12 h
PR mergées (30 j)
5

Description

Running a file info scan on the source of Linux 4.15.7 takes a long time when the disk cache is being used.

Disk cache used (default `--max-in-memory 10000`):

```
$ scancode -i -n4 ~/Desktop/linux-stable-4.15.7 --json ~/Desktop/kernel.json
Setup plugins...
Collect file inventory...
Scan files for: info with 4 process(es)...
[####################] 62061
Scanning done.
Summary: info with 4 process(es)
Errors count: 0
Scan Speed: 65.67 files/sec. 461.51 KB/sec.
Initial counts: 12452 resource(s): 11039 file(s) and 1413 directorie(s)
Final counts: 12452 resource(s): 11039 file(s) and 1413 directorie(s) for 75.76 MB
Timings:
inventory: 1382.95s
scan: 168.11s
output:json: 30.49s
output: 30.49s
total: 1683.44s
```

For comparison, these are the results of having everything in memory (`--max-in-memory 0`):

```
$ scancode -i -n4 ~/Desktop/linux-stable-4.15.7 --json ~/Desktop/kernel.json --max-in-memory 0
Setup plugins...
Collect file inventory...
Scan files for: info with 4 process(es)...
[####################] 62061
Scanning done.
Summary: info with 4 process(es)
Errors count: 0
Scan Speed: 484.66 files/sec. 5.84 MB/sec.
Initial counts: 66438 resource(s): 62061 file(s) and 4377 directorie(s)
Final counts: 66438 resource(s): 62061 file(s) and 4377 directorie(s) for 747.61 MB
Timings:
inventory: 8.80s
scan: 128.05s
output:json: 9.31s
output: 9.31s
total: 147.48s
```

Guide de contribution

Ouvrir le guide de contribution

Évaluation

Cette issue n'a pas encore été évaluée.

Recevez les nouvelles issues par e-mail

Un résumé court des issues GitHub adaptées aux débutants.