opensafely-core / opensafely-core/opencodelists
Vacuum coding system and mapping databases
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 60
- Forks
- 16
- Avg merge
- 4d 12h
- Merged PRs (30d)
- 17
Description
Why are we doing this?
There are a number of SQLite databases on the production server taking up a lot of space. SQL vacuum can potentially reduce the size of a database. The coding system versions and mapping files are represented as distinct write-once SQLite databases. Possibly we can save some space by vacuuming these databases (although possibly not if they are already maximally space-efficient when created), once (as they are never written to after initial creation).
Note https://github.com/opensafely-core/opencodelists/issues/2382 speaks to VACUUMing the core DB automatically on a regular basis.
How will we know when it's done?
We've checked whether doing this saves space and if so applied it to all relevant databases manually or by script.
Possibly include doing this in the scripts that create the databases.
It's okay if the outcome is not doing this at all -- but we should do the investigation.
What are we doing?
As above. Checking should be done with a local download, not live on the production server. If we do think it's worth doing, figure out how to apply it in production.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with a local download of the coding system version and mapping SQLite databases and measure their sizes before and after SQL VACUUM. Inspect the scripts that create these databases to identify all relevant outputs. Done means documenting whether space is saved and, if worthwhile, applying the process manually or through a script without testing on the live production server.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, sqlite
- Domain
- databases
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100