Support auto embed column&expression
- Dominant language
- Go
- Stars
- 40.5k
- Forks
- 6.2k
- PR merge metrics
- PR metrics pending
Description
## Feature Request
**Is your feature request related to a problem? Please describe:**
By providing an API key, you can automatically insert vectors alongside the original content.
**Describe the feature you'd like:**
The workflow is like this:
```
SET @@TIDB_EXP_EMBED_JINA_API_KEY="xxxx";
SET @@TIDB_EXP_EMBED_OPENAI_API_KEY="xxxx";
SET @@TIDB_EXP_EMBED_OPENAI_API_BASE="http://....."; (not supported in Milestone1)
-- Your API Key is transparently encrypted and stored.
-- You can retrieve your API key back by SELECT @@TIDB_EXP_EMBED_JINA_API_KEY.
-- To avoid unauthorized access to your API key, use standalone SQL users to manage the access.
CREATE TABLE t (
content TEXT,
vector VECTOR(..) GENERATED ALWAYS AS
EMBED_TEXT('jina/jina-embeddings-v3', content, ) STORED
);
INSERT INTO t VALUES ('', DEFAULT);
SELECT * FROM t ORDER BY VEC_COSINE_DISTANCE(
content, -- Automatically rewrite to use vector column and embed model name
'query'
) LIMIT 10;
---- The query SQL above will be automatically rewritten into the actual form:
Step1: Expand into TIDB_EMBED
SELECT * FROM t ORDER BY VEC_COSINE_DISTANCE(
vector,
EMBED_TEXT('jina/jina-embeddings-v3', 'query', )
) LIMIT 10;
Step2: Expand into real vector data
SELECT * FROM t ORDER BY VEC_COSINE_DISTANCE(
vector,
[0.5, 0, 0.8, ...]
) LIMIT 10;
```
**Describe alternatives you've considered:**
Ref doc: https://pingcap.feishu.cn/wiki/AYyUwvSAeiHUmBkbx7KcSW8anr9 (pingcap internal only)
**Teachability, Documentation, Adoption, Migration Strategy:**
Contributor guide
Research direction
No source files or tests are named. Start by tracing TiDB's support for generated columns, session variables, vector expressions, and SQL rewriting, then review the CREATE TABLE and query examples in the issue. Done means the documented API-key settings, automatic embedding generation, and both query-rewrite steps work with coverage for the described workflow.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, sql
- Domain
- ai, databases
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100