Allow casting FLOAT_VECTOR(n) to FLOAT_VECTOR(n+x)
- Dominant language
- Java
- Stars
- 4.4k
- Forks
- 616
- Avg merge
- 14h 57m
- Merged PRs (30d)
- 203
Description
### Problem Statement
For storage a column can only have one `float_vector` length.
In some use cases we may have vectors of different lengths.
Since normalized vectors (all values 0 to 1) still compare correctly both with euclidean distance and cosine similarity when padded, we may want to define the table with the maximum expected vector length and pad shorter vectors with zeros on the right on INSERT.
### Possible Solutions
Allow the casting padding with zeroes on the right.
### Considered Alternatives
- padding done by the ingestion process
- ingest done to an `ARRAY(DOUBLE)` field combined with a separate `GENERATED` column where we can do the padding with an UDF
Contributor guide
Research direction
No files, tests, or entry points are named. Start by tracing FLOAT_VECTOR type-casting and INSERT handling, then identify the existing behavior for differing vector lengths; done means casting to a larger length succeeds with zero padding on the right while preserving the stated distance behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, sql
- Domain
- databases
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100