Add Genie Support Plugin

Summary

Genie is Databricks' conversational analytics feature — a natural-language interface that turns a plain-English question into generated SQL and a query result against a curated set of tables. This issue adds Genie tasks to plugin-databricks so a flow can ask a question against a Genie space and get back the generated SQL and result, without a human opening the Databricks UI.

Motivation

AI-driven reporting and ad hoc analytics flows increasingly want to go from "a stakeholder's plain-English question" to "a concrete answer" without a human translating that into SQL by hand. Genie already does that translation inside Databricks; today, using it from an automated pipeline means writing a custom script task against the Genie REST API's async start/poll/fetch-result flow. A native Genie task lets flows — for example, a Slack digest that answers a recurring business question every morning, or an incident-response flow that asks a diagnostic question against operational tables — call Genie the same way they call any other Databricks step, with Kestra handling the polling.

Context

Part of the Databricks platform coverage EPIC: https://github.com/kestra-io/kestra-ee/issues/9384

API Reference

• Official docs: https://docs.databricks.com/api/workspace/genie (usage guide: https://docs.databricks.com/aws/en/genie/conversation-api)
• Authentication: same Databricks workspace auth as the rest of the plugin — OAuth M2M (recommended for automation) or PAT, Bearer token
• Base URL pattern: https://<workspace-instance>/api/2.0/genie/spaces/{space_id}/... — every call is scoped to a pre-existing Genie space (created via the UI or the Management API, configured with a SQL warehouse and curated tables)
• SDK / client library: Databricks SDK for Java, com.databricks:databricks-sdk-java (latest stable: 0.130.0) — WorkspaceClient.genie() returns a GenieAPI (package com.databricks.sdk.service.dashboards) with startConversation(spaceId, content), createMessage(...), getMessage(spaceId, conversationId, messageId), getMessageQueryResult(...), getMessageAttachmentQueryResult(...). startConversation/createMessage return a Wait<GenieMessage, ...> with a blocking .get() that already implements the recommended poll-with-backoff (~1–5s, up to ~1 min) behavior.

Gradle Dependencies

Add to build.gradle:

// Databricks SDK for Java — GenieAPI client, including built-in async polling via Wait<>
implementation "com.databricks:databricks-sdk-java:0.130.0"

Use the latest stable version available on Maven Central.

Kestra framework inclusions (do not add as dependencies): Kestra's internal HTTP client and Jackson serializers are provided by the framework — not needed here since the official Databricks SDK handles the Genie API directly, including its polling.

Plugin Structure

• Repository: plugin-databricks (existing)
• Namespace: io.kestra.plugin.databricks.genie
• Sub-plugins: none (flat package)
• Categories: AI

Task class naming: use concise action names — AskQuestion, Continue — not GenieAskQuestion, AskGenieQuestion, etc.

Suggested Tasks

1. AskQuestion — starts a new Genie conversation with a question (WorkspaceClient.genie().startConversation), blocks on the SDK's Wait<> until the answer is ready (with a configurable timeout), and returns the generated SQL and query result
2. Continue — sends a follow-up question in an existing conversation (createMessage), for multi-turn flows that need to refine a prior answer
3. Surface conversationId/messageId as outputs so a Continue task in the same flow can reference the prior turn
4. Handle the case where Genie returns a text-only answer (no SQL generated) distinctly from a SQL+result answer, and surface both cleanly in outputs
5. Write unit + integration tests (mock the GenieAPI client — no Testcontainers image exists for Genie)
6. Add package-info.java with @PluginSubGroup(category = PluginSubGroup.PluginCategory.AI)
7. Add metadata/genie.yaml and plugin icon SVG
8. Add YAML examples and update io.kestra.plugin.databricks.md with a ## Genie section, noting the Genie space prerequisite

YAML Examples

Note: Genie's API is a synchronous ask-and-answer flow (no events to subscribe to), so no trigger example applies to this sub-plugin — both examples below are tasks.

Example 1 — Ask a question and log the answer

id: genie_ask_question
namespace: company.team

tasks:
  - id: ask_revenue_question
    type: io.kestra.plugin.databricks.genie.AskQuestion
    workspaceHost: "{{ secret('DATABRICKS_HOST') }}"
    token: "{{ secret('DATABRICKS_TOKEN') }}"
    spaceId: "01ef8b2f3a9c1a2d9b7e5f6a1b2c3d4e"
    question: "What was total revenue by region last quarter?"
    timeout: PT2M

  - id: log_answer
    type: io.kestra.plugin.core.log.Log
    message: "Genie SQL: {{ outputs.ask_revenue_question.query }} — Result: {{ outputs.ask_revenue_question.result }}"

Example 2 — Ask a follow-up question in the same conversation

id: genie_follow_up
namespace: company.team

tasks:
  - id: ask_initial
    type: io.kestra.plugin.databricks.genie.AskQuestion
    workspaceHost: "{{ secret('DATABRICKS_HOST') }}"
    token: "{{ secret('DATABRICKS_TOKEN') }}"
    spaceId: "01ef8b2f3a9c1a2d9b7e5f6a1b2c3d4e"
    question: "What was total revenue by region last quarter?"

  - id: ask_followup
    type: io.kestra.plugin.databricks.genie.Continue
    workspaceHost: "{{ secret('DATABRICKS_HOST') }}"
    token: "{{ secret('DATABRICKS_TOKEN') }}"
    spaceId: "01ef8b2f3a9c1a2d9b7e5f6a1b2c3d4e"
    conversationId: "{{ outputs.ask_initial.conversationId }}"
    question: "Now break that down by month."

Acceptance Criteria

Functional

• AskQuestion starts a conversation and returns generated SQL + result once ready
• Continue sends a follow-up message in an existing conversation
• Text-only vs. SQL+result answers both surface cleanly in outputs
• Unit + integration tests pass (./gradlew test)
• Build passes (./gradlew build)

Kestra Plugin Coding Standards

• All new properties use Property<T>
• token annotated with @PluginProperty(secret = true)
• Every property and output has a @Schema annotation
• Task classes carry the five mandatory Lombok annotations (@SuperBuilder, @ToString, @EqualsAndHashCode, @Getter, @NoArgsConstructor)
• Logging via runContext.logger() only
• JSON serialization uses Jackson mappers from io.kestra.core.serializers
• All Property<T> fields support Kestra expression language (template rendering)

Documentation & Structure

• @Plugin(examples = ...) entries each set full = true with a complete runnable flow (id + namespace + tasks)
• Sensitive values in examples use {{ secret('SECRET_NAME') }}
• package-info.java with @PluginSubGroup(category = PluginSubGroup.PluginCategory.AI)
• metadata/genie.yaml and plugin icon SVG present
• io.kestra.plugin.databricks.md updated with a Genie section noting the Genie space prerequisite

---

View as Artifact

Start with build.gradle and existing plugin-databricks task patterns, then inspect the Databricks Java SDK GenieAPI entry points named in the issue. Implement the AskQuestion and Continue tasks under io.kestra.plugin.databricks.genie, add mocked unit and integration tests, and update metadata/genie.yaml, the plugin icon, YAML examples, and io.kestra.plugin.databricks.md. Done means ./gradlew test and ./gradlew build pass and the listed acceptance criteria are met.