feature: in-DB model serving and LLM builtins
- Dominant language
- Go
- Stars
- 40.5k
- Forks
- 6.2k
- PR merge metrics
- PR metrics pending
Description
## Feature Request
**Is your feature request related to a problem? Please describe:**
Users want to run model inference inside TiDB for data-locality, governance, and simplified ops. Today this requires external services and custom glue, which complicates auditing, configuration, and performance tuning.
**Describe the feature you'd like:**
- MODEL DDL to register model metadata and artifacts.
- MODEL_PREDICT execution with ONNX runtime backend.
- MLflow sidecar backend integration and config.
- Governance sysvars (enable/disable, batch size, timeout, null handling, cache settings).
- LLM builtins (llm_complete, llm_embed_text) with provider integrations (Vertex, Bedrock).
- Observability metrics and targeted tests.
**Describe alternatives you've considered:**
- External inference service called from the application layer.
- UDF/plugin-based inference.
These alternatives add operational complexity and don't provide integrated governance or optimizer awareness.
**Teachability, Documentation, Adoption, Migration Strategy:**
- Document MODEL DDL and MODEL_PREDICT usage with examples.
- Document sysvars and config sections for MLflow and LLM providers.
- Keep defaults disabled via sysvars so adoption is opt-in.
**Related PRs:**
- #66594 (core model serving: ONNX/MLflow)
- codex/llm-builtins (LLM builtins/providers; PR to be opened)
Contributor guide
Assessment
This issue has not been assessed yet.