tortoise / tortoise/tortoise-orm

Add vector field (pgvector, sqlite-vss, etc)

Open
#1,467 0 comments 8 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
5.6k
Forks
516
Avg merge
2d 21h
Merged PRs (30d)
9

Description

TL;DR make it possible to use tortoise-orm as a vector database

Is your feature request related to a problem? Please describe.
A lot of commercial vector databases are being thrown around but none of them adequately support your own schemas and never have the ability to use an ORM, pgsql+pgvector is probably the best choice so it would be good to support it in tortoise-orm .

Describe the solution you'd like
Add a new vector field, if using a postgresql database, it uses pgvector and if using sqlite it uses sqlite-vss.
Ideally it would allow adding/updating an index after rows have been added (recreate possible?)

Describe alternatives you've considered
The only real alternative is to not use tortoise-orm

Additional context
Vectors are used to store embeddings, such as those coming from OpenAI, they can be used to retrieve documents based on their semantic similarity, images by content and so on.

Links:
https://towardsdatascience.com/explaining-vector-databases-in-3-levels-of-difficulty-fc392e48ab78
https://platform.openai.com/docs/guides/embeddings
https://github.com/pgvector/pgvector
https://github.com/asg017/sqlite-vss

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reviewing tortoise-orm's field and PostgreSQL/SQLite backend extension points, then compare the pgvector and sqlite-vss requirements named in the issue. Define the vector field's storage and index behavior for both databases, including how indexes are added or recreated. Done means support is tested for both backends.

Written by the indexing model from the issue text.

Assessment

Tech stack
postgresql, python, sqlite
Domain
databases
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.