performance issues when scanning files individually

未关闭

#591 创建于 2017年4月12日 5 条评论 0 个 reaction 已指派 0 人在 GitHub 查看

core and api enhancement

描述

I ran two tests: one on scanning a directory, and one on scanning all the files of the directory separately. The performance difference is quite big.

real 6m57.747s
user 6m2.996s
sys 0m48.011s

when doing:

time find /tmp/bzip2-1.0.1/ -type f -print0 | xargs -0 -n 1 ./bin/scancode

and:

real 1m0.930s
user 0m59.054s
sys 0m1.309s

when scanning the directory normally.

I did some (unprofessional) benchmarks: on my system loading the index pickle from disk takes about 3 seconds.

Maybe other alternatives, like msgpack, might work?

这个 Issue 还没有评估数据。