aboutcode-org / aboutcode-org/scancode-workbench

Suggestion to speed up initial JSON import

Open
#355 0 comments 0 reactions 0 assignees View on GitHub
performance
Dominant language
TypeScript
Stars
173
Forks
81
PR merge metrics
No merged PRs in 30d

Description

### Description
When I import a JSON file, I noted that the sqlite DB file seems to be flushed on disk incrementally and I can see the sqlite file grow as the import goes.
I have an idea: if you import to a memory only database and once this is all imported and only then you save on disk this may speed things up? this may be worth a try

### System configuration
**Operating System**:
Linux Ubuntu 16.04 x86_64

**ScanCode Workbench Version**:
3.0.0b2

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.