aboutcode-org / aboutcode-org/vulnerablecode

Run importers in Parallel

Open
#837 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
702
Forks
328
Avg merge
3d 8h
Merged PRs (30d)
3

Description

Instead of running `./manage.py import --all` in sequence
We can use multiprocessing and try to use all CPU power.

suggestions:
- we can use Celery and schedule tasks and run easily importers in parallel
(another benefit: we can run periodic Tasks and make importers synchronize )

- we can make it simple and use some code from scancode.

this could be a solution for #833

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.