crate / crate/crate

Allow casting FLOAT_VECTOR(n) to FLOAT_VECTOR(n+x)

Open
#17,527 0 comments 0 reactions 0 assignees View on GitHub
feature: sql: type system needs upvotes
Dominant language
Java
Stars
4.4k
Forks
616
Avg merge
14h 57m
Merged PRs (30d)
203

Description

### Problem Statement

For storage a column can only have one `float_vector` length.
In some use cases we may have vectors of different lengths.
Since normalized vectors (all values 0 to 1) still compare correctly both with euclidean distance and cosine similarity when padded, we may want to define the table with the maximum expected vector length and pad shorter vectors with zeros on the right on INSERT.

### Possible Solutions

Allow the casting padding with zeroes on the right.

### Considered Alternatives

- padding done by the ingestion process
- ingest done to an `ARRAY(DOUBLE)` field combined with a separate `GENERATED` column where we can do the padding with an UDF

Contributor guide

Open the contributing guide

Research direction

No files, tests, or entry points are named. Start by tracing FLOAT_VECTOR type-casting and INSERT handling, then identify the existing behavior for differing vector lengths; done means casting to a larger length succeeds with zero padding on the right while preserving the stated distance behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, sql
Domain
databases
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.