simonw / simonw/sqlite-utils

Optional caching mechanism for table.lookup()

Open
#350 3 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

enhancement python-library research
Dominant language
Python
Stars
2.2k
Forks
172
Avg merge
9m
Merged PRs (30d)
1

Description

Inspired by work on git-history where I used this pattern:

    column_name_to_id = {}

    def column_id(column):
        if column not in column_name_to_id:
            id = db["columns"].lookup(
                {"namespace": namespace_id, "name": column},
                foreign_keys=(("namespace", "namespaces", "id"),),
            )
            column_name_to_id[column] = id
        return column_name_to_id[column]

If you're going to be doing a large number of table.lookup(...) calls and you know that no other script will be modifying the database at the same time you can presumably get a big speedup using a Python in-memory cache - maybe even a LRU one to avoid memory bloat.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

The issue names table.lookup() and a Python cache pattern from git-history, but no implementation file or test. Start by locating table.lookup() and its existing tests, then establish the optional cache behavior, including the stated database-modification constraint and possible LRU approach. Done means repeated lookups can use an in-memory cache without unbounded memory growth.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, sqlite
Domain
databases, performance
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.