aboutcode-org / aboutcode-org/scancode-toolkit

Disk cache usage is slow

未关闭
#1,014 3 条评论 0 个 reaction 已指派 0 人 在 GitHub 查看
core and api documentation performance Priority: medium
主要语言
Python
星标
2.6k
派生
791
平均合并
1 天 12 小时
30 天内合并 PR
5

描述

Running a file info scan on the source of Linux 4.15.7 takes a long time when the disk cache is being used.

Disk cache used (default `--max-in-memory 10000`):

```
$ scancode -i -n4 ~/Desktop/linux-stable-4.15.7 --json ~/Desktop/kernel.json
Setup plugins...
Collect file inventory...
Scan files for: info with 4 process(es)...
[####################] 62061
Scanning done.
Summary: info with 4 process(es)
Errors count: 0
Scan Speed: 65.67 files/sec. 461.51 KB/sec.
Initial counts: 12452 resource(s): 11039 file(s) and 1413 directorie(s)
Final counts: 12452 resource(s): 11039 file(s) and 1413 directorie(s) for 75.76 MB
Timings:
inventory: 1382.95s
scan: 168.11s
output:json: 30.49s
output: 30.49s
total: 1683.44s
```

For comparison, these are the results of having everything in memory (`--max-in-memory 0`):

```
$ scancode -i -n4 ~/Desktop/linux-stable-4.15.7 --json ~/Desktop/kernel.json --max-in-memory 0
Setup plugins...
Collect file inventory...
Scan files for: info with 4 process(es)...
[####################] 62061
Scanning done.
Summary: info with 4 process(es)
Errors count: 0
Scan Speed: 484.66 files/sec. 5.84 MB/sec.
Initial counts: 66438 resource(s): 62061 file(s) and 4377 directorie(s)
Final counts: 66438 resource(s): 62061 file(s) and 4377 directorie(s) for 747.61 MB
Timings:
inventory: 8.80s
scan: 128.05s
output:json: 9.31s
output: 9.31s
total: 147.48s
```

贡献指南

打开贡献指南

评估

这个 Issue 还没有评估数据。

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。