pingcap / pingcap/tidb

Support auto embed column&expression

Open
#69,819 0 comments 0 reactions 0 assignees View on GitHub
component/expression sig/execution sig/planner type/feature-request
Dominant language
Go
Stars
40.5k
Forks
6.2k
PR merge metrics
PR metrics pending

Description

## Feature Request

**Is your feature request related to a problem? Please describe:**

By providing an API key, you can automatically insert vectors alongside the original content.

**Describe the feature you'd like:**

The workflow is like this:
```
SET @@TIDB_EXP_EMBED_JINA_API_KEY="xxxx";
SET @@TIDB_EXP_EMBED_OPENAI_API_KEY="xxxx";
SET @@TIDB_EXP_EMBED_OPENAI_API_BASE="http://....."; (not supported in Milestone1)

-- Your API Key is transparently encrypted and stored.
-- You can retrieve your API key back by SELECT @@TIDB_EXP_EMBED_JINA_API_KEY.
-- To avoid unauthorized access to your API key, use standalone SQL users to manage the access.

CREATE TABLE t (
content TEXT,
vector VECTOR(..) GENERATED ALWAYS AS
EMBED_TEXT('jina/jina-embeddings-v3', content, ) STORED
);

INSERT INTO t VALUES ('', DEFAULT);

SELECT * FROM t ORDER BY VEC_COSINE_DISTANCE(
content, -- Automatically rewrite to use vector column and embed model name
'query'
) LIMIT 10;

---- The query SQL above will be automatically rewritten into the actual form:

Step1: Expand into TIDB_EMBED
SELECT * FROM t ORDER BY VEC_COSINE_DISTANCE(
vector,
EMBED_TEXT('jina/jina-embeddings-v3', 'query', )
) LIMIT 10;

Step2: Expand into real vector data
SELECT * FROM t ORDER BY VEC_COSINE_DISTANCE(
vector,
[0.5, 0, 0.8, ...]
) LIMIT 10;
```

**Describe alternatives you've considered:**

Ref doc: https://pingcap.feishu.cn/wiki/AYyUwvSAeiHUmBkbx7KcSW8anr9 (pingcap internal only)

**Teachability, Documentation, Adoption, Migration Strategy:**

Contributor guide

Open the contributing guide

Research direction

No source files or tests are named. Start by tracing TiDB's support for generated columns, session variables, vector expressions, and SQL rewriting, then review the CREATE TABLE and query examples in the issue. Done means the documented API-key settings, automatic embedding generation, and both query-rewrite steps work with coverage for the described workflow.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, sql
Domain
ai, databases
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.