simonw / simonw/llm

Mechanism for recording a different model ID from the one requested

Open
#467 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

design enhancement
Dominant language
Python
Stars
12.5k
Forks
998
Avg merge
3d 13h
Merged PRs (30d)
10

Description

Inspired by this comment in https://simonwillison.net/2024/Apr/22/llama-3/#local-llama-3-70b-instruct-with-llamafile

One warning about this approach: if you use LLM like this then every prompt you run through llamafile will be stored under the same model name in your SQLite logs, even if you try out different llamafile models at different times. You could work around this by registering them with different model_id values in the YAML file.

This made me think: there are other cases where the model that executed the prompt may not be the exact model you specified. Calling OpenAI and asking for gpt-4-turbo and getting a response run against the more recent dated model for example.

It would be useful to have a mechanism for recording the actual model used (as opposed to the model_id requested) in the SQLite logs.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by tracing how the requested model_id reaches the SQLite logging path, using the linked logging documentation for context. Determine where the executed model identity could be recorded separately; the issue names no source files or tests, so repository exploration is required to establish the implementation scope and verification steps.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, sqlite
Domain
databases
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.