apache / apache/incubator-xtable
AI agent tooling for XTable
- Dominant language
- Java
- Stars
- 1.2k
- Forks
- 212
- Avg merge
- 4d 9h
- Merged PRs (30d)
- 16
Description
### Feature Request / Improvement
Umbrella issue for making XTable usable by AI agents, and for deciding where agent-specific code and configuration belong.
The motivation is concrete: getting a first conversion working is still the main friction point for new users. Writing a valid `datasetConfig`, knowing which jars the classpath needs, and interpreting the output are all places people get stuck. An agent can remove most of that, but only if the tooling underneath is legible to it.
## Approach
Three layers, in order. The lower layers are useful on their own, with or without any agent.
1. **An agent-legible CLI.** Structured output, real exit codes, a dry-run mode, explicit no-op reporting. Today `RunSync` throws away the `Map` that `ConversionController.sync()` returns, so the only interface is log text. This layer benefits every caller: Airflow, cron, CI, monitoring, shell scripts, and agents equally.
2. **A machine-readable config contract.** A JSON Schema for `datasetConfig`, generated from the code so it cannot drift from reality.
3. **One tool layer, exposed over MCP.** Vendor-neutral, so it works with any MCP client rather than one product.
Then, separately, a policy question: where harness-specific configuration lives, if anywhere.
## Prior art worth following
`apache/hudi` has answered this question already. They built `hudi-agent-gateway`: one deployable service hosting an agent chat API, an **MCP server**, and a chat UI over a single guarded tool registry. Notable choices:
- One registration exposes a tool to every surface at once, so there is no second API to keep in sync.
- The default LLM provider is Ollama with a local model, so it starts with no API key and endorses no vendor.
- Guardrails are a first-class module, with AST-level checks, row caps and structured invocation logging.
- The test suite runs offline against a fake backend and a scripted model.
Also worth noting what they did **not** do: there is no `CLAUDE.md`, `AGENTS.md`, `SKILL.md`, `.claude/` or `.cursor/` anywhere in `apache/hudi`.
One difference matters for us. Their v1 tools are read-only queries. XTable's central operation **writes metadata**, so a `convert` tool needs a dry-run default and explicit confirmation before a model can call it.
## Sub-issues
See the linked sub-issues for the individual pieces.
## Related
- PR #829, which proposes a Claude Code skill in the repo. Its harness-neutral parts, the config schema, the troubleshooting taxonomy and the verifier, are what layers 1 to 3 are made of.
- #666, the XTable REST service. The MCP surface should sit on the same tool layer rather than becoming a parallel API.
- PR #830, source-format detection, which makes an `inspect` tool nearly free.
### Are you willing to submit PR?
- [x] Yes I am willing to submit a PR!
### Code of Conduct
- [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
Contributor guide
No contributing guide indexed for this repository
Research direction
Review the linked sub-issues, RunSync, and ConversionController.sync(), beginning with how the returned Map is currently handled. Define the boundaries and acceptance criteria for structured CLI output, a generated datasetConfig schema, and the MCP tool layer; completion depends on the individual sub-issues.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, json
- Domain
- api, cli, tooling
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100