aboutcode-org / aboutcode-org/scancode-toolkit

Disk cache usage is slow

Abierto
#1,014 3 comentarios 0 reacciones 0 asignados Ver en GitHub
core and api documentation performance Priority: medium
Lenguaje dominante
Python
Estrellas
2.6k
Forks
791
Merge medio
1 d 12 h
PR fusionados (30 d)
5

Descripción

Running a file info scan on the source of Linux 4.15.7 takes a long time when the disk cache is being used.

Disk cache used (default `--max-in-memory 10000`):

```
$ scancode -i -n4 ~/Desktop/linux-stable-4.15.7 --json ~/Desktop/kernel.json
Setup plugins...
Collect file inventory...
Scan files for: info with 4 process(es)...
[####################] 62061
Scanning done.
Summary: info with 4 process(es)
Errors count: 0
Scan Speed: 65.67 files/sec. 461.51 KB/sec.
Initial counts: 12452 resource(s): 11039 file(s) and 1413 directorie(s)
Final counts: 12452 resource(s): 11039 file(s) and 1413 directorie(s) for 75.76 MB
Timings:
inventory: 1382.95s
scan: 168.11s
output:json: 30.49s
output: 30.49s
total: 1683.44s
```

For comparison, these are the results of having everything in memory (`--max-in-memory 0`):

```
$ scancode -i -n4 ~/Desktop/linux-stable-4.15.7 --json ~/Desktop/kernel.json --max-in-memory 0
Setup plugins...
Collect file inventory...
Scan files for: info with 4 process(es)...
[####################] 62061
Scanning done.
Summary: info with 4 process(es)
Errors count: 0
Scan Speed: 484.66 files/sec. 5.84 MB/sec.
Initial counts: 66438 resource(s): 62061 file(s) and 4377 directorie(s)
Final counts: 66438 resource(s): 62061 file(s) and 4377 directorie(s) for 747.61 MB
Timings:
inventory: 8.80s
scan: 128.05s
output:json: 9.31s
output: 9.31s
total: 147.48s
```

Guía de contribución

Abrir la guía de contribución

Evaluación

Este issue todavía no se ha evaluado.

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.