kestra-io / kestra-io/plugin-databricks

Add Unity Catalog Support Plugin

Open
#250 0 comments 0 reactions 0 assignees View on GitHub
area/plugin
Dominant language
Java
Stars
3
Forks
10
Avg merge
2d 59m
Merged PRs (30d)
9

Description

## Summary

Unity Catalog is now the central governance layer for almost every Databricks workspace — catalogs, schemas, tables, volumes, and grants all live there — yet `plugin-databricks` has no tasks that manage these objects directly. This issue adds a Unity Catalog sub-plugin covering catalog/schema/table/volume/grant management, including the Volumes Files API used to upload and download files (the documented replacement for the legacy DBFS API).

## Motivation

Teams that provision or govern Databricks environments from Kestra currently fall back to ad hoc script tasks calling the Unity Catalog REST API directly, or manage these objects entirely outside of Kestra (Terraform, the Databricks UI, notebooks). Neither keeps catalog/schema/volume provisioning in the same flow as the cluster, job, and SQL steps `plugin-databricks` already orchestrates. Native UC tasks let a single flow, for example, create a schema, provision a volume, upload a dataset into it, and grant access to a downstream team — all with retries, approvals, and notifications, instead of stitching together disconnected scripts.

## Context

Part of the Databricks platform coverage EPIC: https://github.com/kestra-io/kestra-ee/issues/9384

Related but distinct: issue #214 ("Add Zerobus Ingest subplugin for push-based ingestion into Unity Catalog Delta tables") covers *data ingestion into* UC tables, not catalog/schema/table/volume *management* — this issue is scoped to the governance/metadata layer and file I/O, with no overlap.

This issue is also a dependency for the DBFS legacy-notice issue, which will point users to the Volumes file tasks introduced here as the DBFS replacement.

## API Reference

- **Official docs**: https://docs.databricks.com/api/workspace/catalogs (sibling reference pages exist for `schemas`, `tables`, `volumes`, `grants`, and `files` under the same `/api/workspace/` path)
- **Authentication**: same Databricks workspace auth as every other task in this plugin — Bearer token via Personal Access Token (PAT) or OAuth M2M (service principal client-credentials)
- **Base URL pattern**: `https:///api/2.1/unity-catalog/` for catalog/schema/table/volume metadata and grants; `https:///api/2.0/fs/files/` (and `/api/2.0/fs/directories/`) for volume file I/O (the "Files API")
- **SDK / client library**: Databricks SDK for Java, `com.databricks:databricks-sdk-java` (latest stable: `0.130.0`) — `WorkspaceClient` exposes `catalogs()`, `schemas()`, `tables()`, `volumes()`, `grants()` (package `com.databricks.sdk.service.catalog`), and `files()` (package `com.databricks.sdk.service.files`) for volume content upload/download/list

## Gradle Dependencies

Add to `build.gradle`:

```groovy
// Databricks SDK for Java — Unity Catalog and Files API clients
implementation "com.databricks:databricks-sdk-java:0.130.0"
```

> Use the latest stable version available on Maven Central.
>
> **Kestra framework inclusions (do not add as dependencies):** Kestra's internal HTTP client (`io.kestra.core.http.client`) and Jackson serializers are provided by the framework — not needed here since the official Databricks SDK is used directly for all UC/Files API calls.

## Plugin Structure

- **Repository**: `plugin-databricks` (existing)
- **Namespace**: `io.kestra.plugin.databricks.unitycatalog`
- **Sub-plugins**: `catalog`, `schema`, `table`, `volume`, `grant`
- **Categories**: `DATA` (all sub-packages)

> **Task class naming**: task class names must not repeat the plugin or package name as a prefix or suffix. Use concise action/resource names: `Create`, `List`, `Get`, `Update`, `Delete`, `Upload`, `Download`, `Trigger` — not `CatalogCreate`, `CreateCatalog`, etc. The fully-qualified type (`io.kestra.plugin.databricks.unitycatalog.catalog.Create`) already carries the context.
>
> **Gotcha — three-level namespace**: catalog/schema/table/volume objects are addressed by a dot-joined `full_name` (`catalog.schema.table`) or separate `catalogName`/`schemaName`/`name` properties depending on endpoint — keep this consistent across all five sub-packages.
>
> **Gotcha — no `Table.Create`**: the Unity Catalog REST API does not support creating tables directly (tables are created via SQL/Spark, e.g. through the existing `sql.Query` task); scope the `table` sub-package to `List`, `Get`, `Delete` only.
>
> **Gotcha — volume file paths**: `/Volumes////` — the target volume must already exist (via `volume.Create`) before `Upload`/`Download` will succeed.

## Suggested Tasks

1. Implement a shared Unity Catalog connection/auth base class (or reuse the plugin's existing `AbstractTask` + Databricks SDK client builder)
2. `catalog` sub-package — `Create`, `List`, `Get`, `Update`, `Delete`
3. `schema` sub-package — `Create`, `List`, `Get`, `Update`, `Delete`
4. `table` sub-package — `List`, `Get`, `Delete`
5. `volume` sub-package — `Create`, `List`, `Get`, `Update`, `Delete` (metadata) plus `Upload`, `Download` (file content via the Files API)
6. `grant` sub-package — `Get`, `Update` (permissions on a securable, keyed by `securableType` + `fullName`)
7. Add a polling `Trigger` (e.g. under `table`) that fires when a new table appears in a monitored schema
8. Write unit + integration tests
9. Add `package-info.java` per sub-package with `@PluginSubGroup(category = PluginSubGroup.PluginCategory.DATA)`
10. Add `metadata/*.yaml` (one per sub-package) and plugin icon SVGs
11. Add YAML examples and update `io.kestra.plugin.databricks.md` with a `## Unity Catalog` section

## YAML Examples

### Example 1 — Create a schema and a volume, then upload a file into it

```yaml
id: unity_catalog_provision
namespace: company.team

tasks:
- id: create_schema
type: io.kestra.plugin.databricks.unitycatalog.schema.Create
workspaceHost: "{{ secret('DATABRICKS_HOST') }}"
token: "{{ secret('DATABRICKS_TOKEN') }}"
catalogName: "main"
name: "landing_zone"

- id: create_volume
type: io.kestra.plugin.databricks.unitycatalog.volume.Create
workspaceHost: "{{ secret('DATABRICKS_HOST') }}"
token: "{{ secret('DATABRICKS_TOKEN') }}"
catalogName: "main"
schemaName: "landing_zone"
name: "raw_files"
volumeType: MANAGED

- id: upload_file
type: io.kestra.plugin.databricks.unitycatalog.volume.Upload
workspaceHost: "{{ secret('DATABRICKS_HOST') }}"
token: "{{ secret('DATABRICKS_TOKEN') }}"
volumePath: "/Volumes/main/landing_zone/raw_files/data.csv"
from: "{{ outputs.extract.uri }}"
```

### Example 2 — List tables in a schema and grant SELECT to a group

```yaml
id: unity_catalog_grant
namespace: company.team

tasks:
- id: list_tables
type: io.kestra.plugin.databricks.unitycatalog.table.List
workspaceHost: "{{ secret('DATABRICKS_HOST') }}"
token: "{{ secret('DATABRICKS_TOKEN') }}"
catalogName: "main"
schemaName: "landing_zone"

- id: grant_select
type: io.kestra.plugin.databricks.unitycatalog.grant.Update
workspaceHost: "{{ secret('DATABRICKS_HOST') }}"
token: "{{ secret('DATABRICKS_TOKEN') }}"
securableType: SCHEMA
fullName: "main.landing_zone"
changes:
- principal: "data-analysts"
add: ["SELECT"]
```

### Example 3 — React when a new table appears in a schema

```yaml
id: unity_catalog_new_table_trigger
namespace: company.team

triggers:
- id: on_new_table
type: io.kestra.plugin.databricks.unitycatalog.table.Trigger
workspaceHost: "{{ secret('DATABRICKS_HOST') }}"
token: "{{ secret('DATABRICKS_TOKEN') }}"
catalogName: "main"
schemaName: "landing_zone"
interval: PT5M

tasks:
- id: notify
type: io.kestra.plugin.core.log.Log
message: "New table detected: {{ trigger.fullName }}"
```

## Acceptance Criteria

### Functional
- [ ] `catalog`, `schema`, `table`, `volume`, `grant` sub-packages implemented per the CRUD scope above
- [ ] Volume `Upload`/`Download` implemented against the Files API
- [ ] `Trigger` implemented for new-table detection
- [ ] Unit + integration tests pass (`./gradlew test`)
- [ ] Build passes (`./gradlew build`)

### Kestra Plugin Coding Standards
- [ ] All new properties use `Property`
- [ ] `token`/credential properties annotated with `@PluginProperty(secret = true)`
- [ ] Every property and output has a `@Schema` annotation
- [ ] Task classes carry the five mandatory Lombok annotations (`@SuperBuilder`, `@ToString`, `@EqualsAndHashCode`, `@Getter`, `@NoArgsConstructor`)
- [ ] Logging via `runContext.logger()` only
- [ ] JSON serialization uses Jackson mappers from `io.kestra.core.serializers`
- [ ] All `Property` fields support Kestra expression language (template rendering)

### Documentation & Structure
- [ ] `@Plugin(examples = ...)` entries each set `full = true` with a complete runnable flow (id + namespace + tasks/triggers)
- [ ] Sensitive values in examples use `{{ secret('SECRET_NAME') }}`
- [ ] `package-info.java` per sub-package with `@PluginSubGroup(category = PluginSubGroup.PluginCategory.DATA)`
- [ ] `metadata/*.yaml` and plugin icon SVGs present
- [ ] `io.kestra.plugin.databricks.md` updated with a Unity Catalog section

---
*[View as Artifact](https://claude.ai/code/artifact/bdf025e9-7578-4204-81fb-9f1549a97c13)*

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reading the existing plugin's AbstractTask and Databricks SDK client builder, then inspect build.gradle and the requested unitycatalog package structure. Use the listed CRUD, Files API, Trigger, metadata, examples, and documentation requirements as the completion checklist; verify with ./gradlew test and ./gradlew build.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
api, data, databases
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
28/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.