performance issues when scanning files individually

I ran two tests: one on scanning a directory, and one on scanning all the files of the directory separately. The performance difference is quite big.

real 6m57.747s
user 6m2.996s
sys 0m48.011s

when doing:

time find /tmp/bzip2-1.0.1/ -type f -print0 | xargs -0 -n 1 ./bin/scancode

and:

real 1m0.930s
user 0m59.054s
sys 0m1.309s

when scanning the directory normally.

I did some (unprofessional) benchmarks: on my system loading the index pickle from disk takes about 3 seconds.

Maybe other alternatives, like msgpack, might work?