Support for embedding modes / input types
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 12.5k
- Forks
- 998
- Avg merge
- 3d 13h
- Merged PRs (30d)
- 10
Description
Several embedding models supported by LLM plugins have a concept of "modes" - usually called something like "task types" or "input types".
Some examples:
- Gemini: https://github.com/google-gemini/cookbook/blob/9eb52260f979aa8339e6d1eb77323faa178bbd78/quickstarts/rest/Embeddings_REST.ipynb - "Use task_type to provide a hint to the model how you'll use the embeddings"
- Nomic: https://github.com/simonw/llm-nomic-api-embed/issues/2
- E5-large-v2: https://til.simonwillison.net/llms/embed-paragraphs - this one works using
passage:andquery:prefixes.
We need a mechanism to support these in LLM core itself, mainly for the llm similar command - we need to calculate the original stored embeddings for RETRIEVAL_DOCUMENT (in Gemini's terminology) but the search query should be RETRIEVAL_QUERY.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the llm similar command and the embedding-model integrations described in the issue. Compare how Gemini, Nomic, and E5-large-v2 represent document and query inputs, then define what a core mechanism must support; done means stored document embeddings and search-query embeddings can use their respective modes.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- ai, cli
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 28/100