opensafely-core / opensafely-core/opencodelists

Vacuum coding system and mapping databases

Open
#2,383 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

deck-scrubbing enhancement infrastructure
Dominant language
Python
Stars
60
Forks
16
Avg merge
4d 12h
Merged PRs (30d)
17

Description

Why are we doing this?

There are a number of SQLite databases on the production server taking up a lot of space. SQL vacuum can potentially reduce the size of a database. The coding system versions and mapping files are represented as distinct write-once SQLite databases. Possibly we can save some space by vacuuming these databases (although possibly not if they are already maximally space-efficient when created), once (as they are never written to after initial creation).

Note https://github.com/opensafely-core/opencodelists/issues/2382 speaks to VACUUMing the core DB automatically on a regular basis.

How will we know when it's done?

We've checked whether doing this saves space and if so applied it to all relevant databases manually or by script.

Possibly include doing this in the scripts that create the databases.

It's okay if the outcome is not doing this at all -- but we should do the investigation.

What are we doing?

As above. Checking should be done with a local download, not live on the production server. If we do think it's worth doing, figure out how to apply it in production.


Defining delivery tasks guidance

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with a local download of the coding system version and mapping SQLite databases and measure their sizes before and after SQL VACUUM. Inspect the scripts that create these databases to identify all relevant outputs. Done means documenting whether space is saved and, if worthwhile, applying the process manually or through a script without testing on the live production server.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, sqlite
Domain
databases
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.