[FEATURE][UMBRELLA] Data Agent Engine — isolation, identity, and security hardening
- Dominant language
- Scala
- Stars
- 2.4k
- Forks
- 1k
- PR merge metrics
- No merged PRs in 30d
Description
### Code of Conduct
- [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
### Search before asking
- [x] I have searched in the [issues](https://github.com/apache/kyuubi/issues?q=is%3Aissue) and found no similar issues.
### Describe the feature
Umbrella for the security-hardening / isolation follow-ups deferred while landing the Data Agent engine (#7379, last PR #7431). Sub-tasks are tracked inline, not split into separate issues.
**Root cause behind most of these:** `DataAgentProcessBuilder extends ProcBuilder`, so the engine runs as a local child process on the Kyuubi server host (no k8s/YARN launch path) and shares the server's Kerberos credential cache, network identity, and filesystem. That sharing is why Kerberos is currently banned and why credentials must go in the JDBC URL. Giving the engine its own isolated runtime + identity unblocks the rest.
### Sub-tasks
- [ ] **①a Cross-server Data Agent session routing.** Make Kyuubi servers stateless proxies for Data Agent REST sessions. Register a discovery mapping from the external session UUID to its Data Agent engine, then let any server attach to the existing engine-side session handle instead of depending on the creating server's in-memory `KyuubiSessionImpl`. The conversation, approval, and tool state remains in the engine. This fixes requests being routed to another server in a multi-server deployment; it does not preserve a local child engine when its launching server fails.
- [ ] **①b Isolated runtime with its own identity (keystone).** Add a remote launch path so `DATA_AGENT` runs in its own container with a distinct identity instead of as a local child of the server — k8s first, YARN later; keep local-process as a dev fallback. This decouples engine lifetime from a server instance and unblocks ② and ③.
- [ ] **② Kerberos support** *(depends on ①b)*. Engine logs in with its own principal/keytab and impersonates the session user via `hadoop.proxyuser` ACLs, lifting the current ban. Removes the two TODOs in `EngineRef#deriveDataAgentJdbcUrl` and `ChatCompletionProvider`.
- [ ] **③ Credential propagation** *(design depends on ①b)*. Dedicated config / UI inputs + a `propagateCredential` opt-in so secrets aren't embedded in `kyuubi.engine.data.agent.jdbc.url`. Align with the JDBC engine's `propagateCredential` and Trino's `connection.user` fallback. Raised in #7431.
- [ ] **④ Cheaper summarization model for context compaction** *(independent)*. Let `CompactionMiddleware` use a model distinct from the main agent model, falling back when unset (see its TODO).
- [ ] **⑤ Proper SQL read-only check for `run_select_query`** — tracked in #7412; linked for visibility, orthogonal to the above.
**Dependency order:** ①a fixes cross-server request routing first. ①b then decouples engine lifetime and identity from the launching server; ② and ③ build on ①b. ④ and #7412 are independent.
### Are you willing to submit PR?
- [x] Yes. I would be willing to submit a PR with guidance from the Kyuubi community to improve.
Contributor guide
Research direction
This is an umbrella rather than a single change; begin by choosing one sub-task and reading DataAgentProcessBuilder and ProcBuilder, plus the referenced EngineRef#deriveDataAgentJdbcUrl, ChatCompletionProvider, or CompactionMiddleware TODO. Check the dependency order and linked issues before defining a smaller scope. Done means the selected sub-task meets the behavior described in its section and its relevant verification passes.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- hadoop, kubernetes, scala
- Domain
- backend, cloud, distributed-systems, security
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100