simonw / simonw/llm

Rethink model.key= pattern to better support shared instances

Open
#299 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

design enhancement
Dominant language
Python
Stars
12.5k
Forks
991
Avg merge
3d 13h
Merged PRs (30d)
10

Description

No wait! This won't work, because the model instance is shared across multiple calls - so one user setting the key correctly could leak through to another user.

Originally posted by @simonw in https://github.com/simonw/datasette-llm-embed/issues/3#issuecomment-1751915001

Noticed this while trying to add an API key feature to the datasette-llm-embed plugin.

Since a single embedding model instance is created and then shared by all code that calls llm.get_embedding_model(model_id) - which is good for in-memory models - code that sets model.key = ... is not compatible with shared usage environments such as Datasette plugins where we don't want keys to leak to other users.

So need to reconsider that model.key = ... pattern entirely.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by tracing get_embedding_model(model_id) and the shared model.key usage described in the issue. Define a design where API keys supplied by one caller cannot leak through a shared embedding model instance, then verify the behavior across multiple callers and the Datasette plugin use case.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
ai, backend-api-design
Issue type
Refactor
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.