aboutcode-org / aboutcode-org/scancode-toolkit

Disk cache usage is slow

Đang mở
#1,014 3 bình luận 0 reaction 0 người được giao Xem trên GitHub
core and api documentation performance Priority: medium
Ngôn ngữ chính
Python
Star
2.6k
Fork
791
Merge trung bình
1 ngày 12 giờ
Pull request đã merge (30 ngày)
5

Mô tả

Running a file info scan on the source of Linux 4.15.7 takes a long time when the disk cache is being used.

Disk cache used (default `--max-in-memory 10000`):

```
$ scancode -i -n4 ~/Desktop/linux-stable-4.15.7 --json ~/Desktop/kernel.json
Setup plugins...
Collect file inventory...
Scan files for: info with 4 process(es)...
[####################] 62061
Scanning done.
Summary: info with 4 process(es)
Errors count: 0
Scan Speed: 65.67 files/sec. 461.51 KB/sec.
Initial counts: 12452 resource(s): 11039 file(s) and 1413 directorie(s)
Final counts: 12452 resource(s): 11039 file(s) and 1413 directorie(s) for 75.76 MB
Timings:
inventory: 1382.95s
scan: 168.11s
output:json: 30.49s
output: 30.49s
total: 1683.44s
```

For comparison, these are the results of having everything in memory (`--max-in-memory 0`):

```
$ scancode -i -n4 ~/Desktop/linux-stable-4.15.7 --json ~/Desktop/kernel.json --max-in-memory 0
Setup plugins...
Collect file inventory...
Scan files for: info with 4 process(es)...
[####################] 62061
Scanning done.
Summary: info with 4 process(es)
Errors count: 0
Scan Speed: 484.66 files/sec. 5.84 MB/sec.
Initial counts: 66438 resource(s): 62061 file(s) and 4377 directorie(s)
Final counts: 66438 resource(s): 62061 file(s) and 4377 directorie(s) for 747.61 MB
Timings:
inventory: 8.80s
scan: 128.05s
output:json: 9.31s
output: 9.31s
total: 147.48s
```

Hướng dẫn đóng góp

Mở hướng dẫn đóng góp

Đánh giá

Issue này chưa được đánh giá.

Nhận issue mới trong hộp thư của bạn

Bản tóm tắt ngắn những issue GitHub phù hợp với người mới.