Agentic lakehouse: built-in AI agent serving and operations for Hudi
- Dominant language
- Java
- Stars
- 6.2k
- Forks
- 2.5k
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 111
Description
### Feature Description
**What the feature achieves:**
Every Hudi table becomes something you can talk to. This feature adds an optional, self-hostable serving layer to the project: a set of Helm charts (`hudi-lakehouse`) that deploy Trino with the Hudi connector alongside a new `hudi-agent-gateway` service — a single process that hosts an LLM agent loop with guarded, read-only SQL tools over the lakehouse, exposes the same tools to external agents via MCP, and ships a first-party chat UI. Users ask questions in natural language; the agent inspects the catalog, writes SQL, runs it through Trino, and answers grounded in real query results.
**Why this feature is needed:**
Hudi has always been more than a table format — self-managing table services, ingestion utilities, catalog syncing and a CLI make it a data platform. AI-assisted access and operations are the natural next surface of that platform, and there is real value in the project shipping it first-party rather than leaving every team to assemble and glue themselves: the tools stay engine-native (catalog introspection, guarded SQL, and later table-services operations), the whole stack is open and self-hostable with local inference (Ollama) as the out-of-box default and pluggable model providers (vLLM, OpenAI, Anthropic), and answer quality can build on Hudi-specific knowledge no generic text-to-SQL layer has. The longer arc is agents that not only answer questions but help operate tables — tuning, inspecting, and maintaining them.
This issue is the umbrella; the linked sub-issues break the work down, and an initial implementation is ready to land as the first PR.
### User Experience
**How users will use this feature:**
- `helm install` the `hudi-lakehouse` charts against an existing metastore and object store (or run the included one-command minikube environment); the gateway comes up next to Trino.
- Chat UI at the gateway's `/ui/` for interactive analysis and built-in chat; `POST /v1/chat` (JSON or SSE streaming) for programmatic use; `/mcp` so tools like Claude can operate on the lakehouse directly; a Connect panel surfaces every endpoint.
- Inference provider is a values-file choice: Ollama by default, Anthropic or OpenAI via an API key Secret, vLLM as an optional part of the same install, or any OpenAI-compatible URL. The UI adapts to whatever the cluster runs.
- All agent SQL is guarded: parsed, read-only-verified, row-capped, and time-limited; every tool invocation is logged.
- No changes to existing Hudi APIs, configs, or storage — everything is new, optional, and additive.
### Hudi RFC Requirements
**RFC PR link:** N/A
**Why RFC is/isn't needed:**
- Does this change public interfaces/APIs? No
- Does this change storage format? No
- Justification: purely additive, optional modules (charts + a Python service); nothing existing changes behavior. Direction and design are open for community input on a GitHub Discussion to follow (mirrored to dev@). Sub-efforts that do touch engine or format surfaces (e.g. vector search through the SQL endpoint) will get their own RFC/design before work starts.
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.