Adding support for mcp for hudi cli operations
- Dominant language
- Java
- Stars
- 6.2k
- Forks
- 2.5k
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 111
Description
### Describe the problem you faced
Add an MCP (Model Context Protocol) server that exposes Apache Hudi CLI operations as structured tools for AI assistants. This enables LLMs like Claude, ChatGPT, Copilot, and
others to inspect, diagnose, and operate on Hudi tables through natural language.
Related discussion: https://github.com/apache/hudi/discussions/18324
### Motivation
Hudi's CLI is powerful but has a steep learning curve — users need to know exact command syntax, flag names, and safe operation ordering. An MCP server bridges this gap by:
1. **Lowering the barrier to entry** — users describe what they want in plain English; the AI assistant translates to the correct CLI commands
2. **Structured output** — raw ASCII FlipTable output is parsed into JSON, making it reliable for programmatic consumption
3. **Safety by default** — destructive operations (rollbacks, cleaning, compaction) go through a tiered confirmation protocol, preventing accidental data loss
4. **Composability** — multi-step diagnostic workflows (health checks, storage analysis, guided rollbacks) are bundled into single tools
### Scope
#### Tools (40+)
**Connection Management**
- `connect_to_table` — connect to a Hudi table (local, S3, GCS, HDFS, Azure)
- `disconnect` — disconnect from current table
- `show_connection` — show connection status
**Read-Only Execution**
- `execute_hudi_command` — execute any single read-only CLI command
- `execute_hudi_commands` — batch execute multiple read-only commands
**Read-Only Workflows**
- `table_overview` — table description + partitions
- `table_health_check` — comprehensive: desc, commits, cleans, compactions, stats
- `timeline_summary` — active timeline + incomplete instants
- `commit_details` — partition/file/write-stat details for a specific commit
- `storage_analysis` — file sizes, filesystem view, metadata stats
**Write Operations (tiered safety)**
| Risk | Tools | Behavior |
|------|-------|----------|
| LOW | `create_savepoint`, `toggle_lock_audit`, `schedule_compaction`, `schedule_clustering` | Execute immediately |
| MEDIUM | `unschedule_compaction`, `delete_savepoint`, `delete_markers`, `manage_metadata`, `trigger_archival`, `repair_table`, `recover_table_configs`, `update_table_configs`,
`delete_table_configs`, `upgrade_or_downgrade_table` | Require token confirmation |
| HIGH | `rollback_commit`, `rollback_to_savepoint`, `run_compaction`, `run_clustering`, `run_clean` | Dry-run preview + token confirmation |
**Confirmation Protocol**
- `confirm_operation` — execute a pending operation by token
- `cancel_operation` — cancel a pending operation
- `list_pending_operations` — list all pending operations
**Guided Write Workflows**
- `compaction_workflow` — show pending compactions, then prepare execution
- `clustering_workflow` — analyze storage, then prepare clustering
- `safe_rollback_workflow` — show commit impact, then prepare rollback
- `table_repair_workflow` — dry-run repair, then prepare real execution
#### Core Components
| Component | Description |
|-----------|-------------|
| **Command validator** | Allowlist-based read/write separation with 86 read-only and 19 write command prefixes |
| **CLI executor** | Subprocess-based execution of `hudi-cli script --file` |
| **FlipTable parser** | Parses Hudi CLI ASCII table output into structured JSON (`ParsedTable`/`ParsedOutput`) |
| **Safety manager** | Token-based confirmation with 5-minute TTL, single-use tokens |
| **Session manager** | Tracks connected table path, auto-prepends connect commands |
#### Safety Model
```
LOW risk --> Execute immediately, return result
MEDIUM risk --> Generate confirmation token (5-min TTL), return token + description
HIGH risk --> Run dry-run/preview, generate token, return preview + token + description
```
### Technical Details
- **Language**: Python 3.10+
- **Framework**: FastMCP 2.0+
- **Transport**: stdio (compatible with all MCP clients)
- **Dependencies**: `fastmcp>=2.0.0` (single dependency)
- **Location**: `scripts/hudi-cli-mcp/`
- **Requires**: Spark, `hudi-cli-bundle` JAR, `hudi-spark-bundle` JAR
### Supported MCP Clients
- Claude Desktop
- Claude Code (CLI)
- VS Code with Copilot MCP extensions
- Any MCP-compatible client
### Tasks
- [x] Core CLI executor with subprocess management
- [x] FlipTable ASCII output parser
- [x] Command validation and risk classification
- [x] Session management (connect/disconnect)
- [x] Token-based safety manager
- [x] Read-only tools (generic + workflows)
- [ ] Write operation tools (19 operations across 3 risk tiers)
- [ ] Guided write workflows (compaction, clustering, rollback, repair)
- [x] Confirmation protocol (confirm/cancel/list)
- [x] Unit test suite
- [x] README with setup, configuration, and usage docs
- [ ] Website documentation update
- [ ] Support for additional transports (SSE, HTTP)
- [ ] Integration tests with a live Hudi table
### Expected behavior
AI assistants like claude, etc should be able to assist users to manage and operate Hudi using natural language.
### Environment Description
* Hudi version:
* Spark version:
* Flink version:
* Hive version:
* Hadoop version:
* Storage (HDFS/S3/GCS..):
* Running on Docker? (yes/no):
### Additional context
_No response_
### Stacktrace
```shell
```
Contributor guide
No contributing guide indexed for this repository
Research direction
Start in scripts/hudi-cli-mcp/ and review the existing README and unit test suite to understand the implemented executor, parser, validation, safety, and session components. The remaining scope includes 19 write-operation tools, guided write workflows, website documentation, additional transports, and live-table integration tests; done means the selected scope is implemented and covered by appropriate tests.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- ai, cli
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100