apache / apache/hudi

Agentic lakehouse: built-in AI agent serving and operations for Hudi

Open
#19,256 0 comments 0 reactions 1 assignee Claimed by @vinothchandar View on GitHub
type:feature
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

### Feature Description

**What the feature achieves:**

Every Hudi table becomes something you can talk to. This feature adds an optional, self-hostable serving layer to the project: a set of Helm charts (`hudi-lakehouse`) that deploy Trino with the Hudi connector alongside a new `hudi-agent-gateway` service — a single process that hosts an LLM agent loop with guarded, read-only SQL tools over the lakehouse, exposes the same tools to external agents via MCP, and ships a first-party chat UI. Users ask questions in natural language; the agent inspects the catalog, writes SQL, runs it through Trino, and answers grounded in real query results.

**Why this feature is needed:**

Hudi has always been more than a table format — self-managing table services, ingestion utilities, catalog syncing and a CLI make it a data platform. AI-assisted access and operations are the natural next surface of that platform, and there is real value in the project shipping it first-party rather than leaving every team to assemble and glue themselves: the tools stay engine-native (catalog introspection, guarded SQL, and later table-services operations), the whole stack is open and self-hostable with local inference (Ollama) as the out-of-box default and pluggable model providers (vLLM, OpenAI, Anthropic), and answer quality can build on Hudi-specific knowledge no generic text-to-SQL layer has. The longer arc is agents that not only answer questions but help operate tables — tuning, inspecting, and maintaining them.

This issue is the umbrella; the linked sub-issues break the work down, and an initial implementation is ready to land as the first PR.

### User Experience

**How users will use this feature:**
- `helm install` the `hudi-lakehouse` charts against an existing metastore and object store (or run the included one-command minikube environment); the gateway comes up next to Trino.
- Chat UI at the gateway's `/ui/` for interactive analysis and built-in chat; `POST /v1/chat` (JSON or SSE streaming) for programmatic use; `/mcp` so tools like Claude can operate on the lakehouse directly; a Connect panel surfaces every endpoint.
- Inference provider is a values-file choice: Ollama by default, Anthropic or OpenAI via an API key Secret, vLLM as an optional part of the same install, or any OpenAI-compatible URL. The UI adapts to whatever the cluster runs.
- All agent SQL is guarded: parsed, read-only-verified, row-capped, and time-limited; every tool invocation is logged.
- No changes to existing Hudi APIs, configs, or storage — everything is new, optional, and additive.

### Hudi RFC Requirements

**RFC PR link:** N/A

**Why RFC is/isn't needed:**
- Does this change public interfaces/APIs? No
- Does this change storage format? No
- Justification: purely additive, optional modules (charts + a Python service); nothing existing changes behavior. Direction and design are open for community input on a GitHub Discussion to follow (mirrored to dev@). Sub-efforts that do touch engine or format surfaces (e.g. vector search through the SQL endpoint) will get their own RFC/design before work starts.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.