tursodatabase / tursodatabase/libsql

Unable to insert vector data libsql

Open
#1,903 6 comments 2 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
C
Stars
17.2k
Forks
531
Avg merge
1h 12m
Merged PRs (30d)
1

Description

Bug report

I'm unable to persist a vector column with embeddings. I followed the turso guide here here for defining the types And the Drizzle documentation. I've also taken inspiration from this blog post for the setup with expo.

I've open an issue here https://github.com/drizzle-team/drizzle-orm/issues/3899

Expected behaviour

I can insert a vector into my sqlite database

Libs and versions

"@libsql/client": "^0.14.0"
"expo": "^52.0.23"
"expo-sqlite": "~15.0.5"
"drizzle-orm": "^0.38.3"
@op-engineering/op-sqlite": "^11.2.12

Steps
  1. Database configuration + Drizzle config
// db/index.ts

import { drizzle, OPSQLiteDatabase } from "drizzle-orm/op-sqlite";
import {
  open,
  DB
} from "@op-engineering/op-sqlite";
import { logError } from "~/lib/utils";

let db: OPSQLiteDatabase
let sqlite: DB

try { 
  sqlite = open({
    name: "meadowmind.local.db",
  });
  
  db = drizzle(sqlite);

} catch (error) {
  const updatedError = logError('Error initializing database client', error);
  throw updatedError;
}


export { db }

// drizzle.config.ts

import type { Config } from "drizzle-kit";
 
export default {
  schema: "./db/schema.ts",
  out: "./db/generated",
  dialect: "sqlite",
  driver: 'expo',
  casing: "snake_case"
} satisfies Config;
  1. Define custom vector type
const embedding = customType<{
  data: number[];
  config: { dimensions: number };
  configRequired: true;
  driverData: Buffer;
}>({
  dataType(config) {
    return `F32_BLOB(${config.dimensions})`;
  },
  fromDriver(value: Buffer) {
    return Array.from(new Float32Array(value.buffer));
  },
  toDriver(value: number[]) {
    return sql`vector32(${JSON.stringify(value)})`;
  },
});
  1. Define table with vector column
export const content = sqliteTable('content', {
  id: integer('id').primaryKey({ autoIncrement: true }),
title: text('title').notNull(),
  data: text('data', { mode: 'json'}).notNull().$type<{value: string, type: string, extension: string}>(),
  embedding: embedding('embedding', { dimensions: 512 })
});
  1. Add an index in my generated/migrations .sql file
-- Custom SQL migration file, put your code below! --
CREATE INDEX IF NOT EXISTS embedding_index
ON content(embedding)
USING vector_cosine(512)
  1. Run migration to add index
npm run db:migrate
  1. Loading migrations in my _layout.tsx
import { useMigrations } from 'drizzle-orm/op-sqlite/migrator';

const { success, error } = useMigrations(db, migrations);
  1. Insert vector data
import {content } from "../schema";

 await tx.insert(content).values([{
                title,
                data,
                embedding: sql`vector32(${JSON.stringify(embedding)})`
            }])

Observation

In the insert operation in 6) title and data are stored just alright; however, the embedding field is an empty array.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Reproduce the report using the shown Expo/op-sqlite setup, beginning with db/index.ts, drizzle.config.ts, and the custom embedding type in the schema snippet. Compare the inserted vector with the value returned by the driver, and check the linked Drizzle issue before deciding where the problem belongs. Done means the documented migration and insert steps return the stored vector rather than an empty array.

Written by the indexing model from the issue text.

Assessment

Tech stack
sqlite, typescript
Domain
database
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.