simonw / simonw/sqlite-utils

How to insert the vector embeddings values into a table?

Open
#651 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
2.2k
Forks
172
Avg merge
9m
Merged PRs (30d)
1

Description

Reading your blog post on using sqlite-vec for working with embeddings in sqlite, I was trying to do something similar but I'm stuck in the step before all these. I'm not able to insert the embedding values in the table.

Here is what I have tried:

from sqlite_utils import Database
from sqlite_vec import serialize_float32

db_test = Database("test.db")
items = [
    {"id": 1, "text": "hello world"},
    {"id": 2, "text": "hello globe"},
    {"id": 3, "text": "hi world"}
]
db_test["table"].insert_all(items, pk="id")
db_test["table"].add_column("embedding", "BLOB")

embedding = [0.1, 0.2, 0.3, 0.4] ## simplification of getting the embeddings from openai or a local embedding model
db_test["table"].update({"id": 1, "embedding": np.array(item_embedding).tobytes()}, alter=True)

error:

InterfaceError                            Traceback (most recent call last)
Cell In[27], line 1
----> 1 db_test["table"].update({"id": 1, "embedding": np.array(item_embedding).tobytes()}, alter=True)

File ~/.pyenv/versions/workbench/lib/python3.10/site-packages/sqlite_utils/db.py:2798, in Table.update(self, pk_values, updates, alter, conversions)
   2796     pk_values = [pk_values]
   2797 # Soundness check that the record exists (raises error if not):
-> 2798 self.get(pk_values)
   2799 if not updates:
   2800     return self

File ~/.pyenv/versions/workbench/lib/python3.10/site-packages/sqlite_utils/db.py:1555, in Table.get(self, pk_values)
   1553 rows = self.rows_where(" and ".join(wheres), pk_values)
   1554 try:
-> 1555     row = list(rows)[0]
   1556     self.last_pk = last_pk
   1557     return row

File ~/.pyenv/versions/workbench/lib/python3.10/site-packages/sqlite_utils/db.py:1364, in Queryable.rows_where(self, where, where_args, order_by, select, limit, offset)
   1362 if offset is not None:
   1363     sql += " offset {}".format(offset)
-> 1364 cursor = self.db.execute(sql, where_args or [])
   1365 columns = [c[0] for c in cursor.description]
   1366 for row in cursor:
...
--> 533     return self.conn.execute(sql, parameters)
    534 else:
    535     return self.conn.execute(sql)

InterfaceError: Error binding parameter 0 - probably unsupported type.

Other things I tried instead of np.array(embedding).tobytes():

  • db_test["table"].update({"id": 1, "embedding": serialize_float32(embedding)}, alter=True) : same error
  • inserting the embeddings as string instead of blob: same error

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Reproduce the example in the issue using sqlite_utils and inspect db.py around Table.update, Table.get, and rows_where. Trace which parameter reaches SQLite binding, then establish the supported way to store the embedding value and verify the result against the reported error.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
database
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.