kestra-io / kestra-io/plugin-databricks

Mark DBFS tasks as legacy in favor of Unity Catalog Volumes

Open Beginner friendly
#251 1 comment 0 reactions 0 assignees View on GitHub
area/docs area/plugin
Dominant language
Java
Stars
3
Forks
10
Avg merge
2d 59m
Merged PRs (30d)
9

Description

## Summary

Databricks now documents DBFS root and DBFS mounts as deprecated and not recommended, steering users toward Unity Catalog volumes, external locations, or workspace files instead. This issue marks `plugin-databricks`'s existing `dbfs.Upload`/`dbfs.Download` tasks as **legacy** in their documentation and `@Schema` descriptions, pointing users to the new Unity Catalog Volumes tasks (#250) — without removing or breaking the existing tasks.

## Motivation

New Databricks workspaces are already provisioned without DBFS root/mount access by default, and Databricks' own "Best practices for DBFS and Unity Catalog" guidance is unambiguous about steering customers away from DBFS. Users building new flows against `plugin-databricks` today have no signal, inside Kestra itself, that they should reach for Unity Catalog Volumes instead of `dbfs.Upload`/`dbfs.Download` — they'd only discover this by separately reading Databricks' docs. A clear in-product legacy notice (docs + `@Schema` wording) closes that gap cheaply, while intentionally stopping short of a hard Java `@Deprecated` or removal, since DBFS and UC Volumes are documented to coexist and many existing flows depend on the current tasks working unchanged.

## Context

Part of the Databricks platform coverage EPIC: https://github.com/kestra-io/kestra-ee/issues/9384

Depends on #250 ("Add Unity Catalog Support Plugin") — the `unitycatalog.volume` `Upload`/`Download` tasks introduced there are what this issue's documentation points users toward. This issue should not be started until #250's Volumes tasks exist (or at least are far enough along that the doc cross-links resolve).

Databricks' own guidance:
- https://docs.databricks.com/aws/en/dbfs/unity-catalog — "Best practices for DBFS and Unity Catalog" (explicit deprecation language)
- https://docs.databricks.com/aws/en/dbfs — "What is DBFS?"
- https://docs.databricks.com/aws/en/dbfs/dbfs-root — recommendations for DBFS root

## API Reference

- **Official docs**: https://docs.databricks.com/aws/en/dbfs/unity-catalog
- **Authentication**: unchanged — this issue makes no API or auth changes, only documentation/schema wording
- **Base URL pattern**: unchanged (`/api/2.0/dbfs/...` for the existing tasks; no code path changes)
- **SDK / client library**: none — no new dependency

## Gradle Dependencies

None. This issue only touches `@Schema` descriptions and the plugin how-to doc — no new dependency, no behavior change, no version bump beyond what #250 already introduces.

## Plugin Structure

- **Repository**: `plugin-databricks` (existing)
- **Namespace**: `io.kestra.plugin.databricks.dbfs` (existing — no new sub-package)
- **Sub-plugins**: none (documentation-only change to the existing `dbfs` sub-package)
- **Categories**: `DATA` (unchanged)

## Suggested Tasks

1. Update `dbfs.Upload` and `dbfs.Download`'s class-level `@Schema(title = ...)` / description to note that Databricks considers DBFS legacy and to recommend the new `unitycatalog.volume.Upload`/`Download` tasks (from #250) for new flows — wording only, **do not** add a Java `@Deprecated` annotation (would surface as a compiler/IDE warning on every existing flow using these tasks, which is not the intent here)
2. Update `io.kestra.plugin.databricks.md`'s `## Tasks` section for the `dbfs` sub-group with a short callout: DBFS is legacy per Databricks' own guidance, link to the Unity Catalog Volumes tasks as the recommended path for new flows
3. Add a one-line note to `README.md` if it separately documents the `dbfs` tasks
4. No test changes required — behavior is unchanged; existing `dbfs` tests must continue to pass as-is

## YAML Examples

> Note: this issue does not add a new task or trigger type, so the two examples below show the *migration path* (existing DBFS flow vs. its Unity Catalog Volumes equivalent) rather than a novel capability. No trigger example applies — neither `dbfs` nor this issue touches a trigger.

### Example 1 — Existing DBFS upload flow (still supported, now documented as legacy)

```yaml
id: dbfs_upload_legacy
namespace: company.team

tasks:
- id: upload_report
type: io.kestra.plugin.databricks.dbfs.Upload
host: "{{ secret('DATABRICKS_HOST') }}"
token: "{{ secret('DATABRICKS_TOKEN') }}"
from: "{{ outputs.generate_report.uri }}"
to: "dbfs:/reports/daily_report.csv"
```

### Example 2 — Recommended Unity Catalog Volumes equivalent

```yaml
id: uc_volume_upload_recommended
namespace: company.team

tasks:
- id: upload_report
type: io.kestra.plugin.databricks.unitycatalog.volume.Upload
workspaceHost: "{{ secret('DATABRICKS_HOST') }}"
token: "{{ secret('DATABRICKS_TOKEN') }}"
from: "{{ outputs.generate_report.uri }}"
volumePath: "/Volumes/main/reports/daily/daily_report.csv"
```

## Acceptance Criteria

### Functional
- [ ] `dbfs.Upload`/`dbfs.Download` behavior is completely unchanged — no `@Deprecated` annotation added
- [ ] `@Schema` descriptions updated with legacy wording and a link to the Unity Catalog Volumes replacement tasks
- [ ] Existing `dbfs` unit/integration tests continue to pass unmodified

### Kestra Plugin Coding Standards
- [ ] No new properties introduced — this is a documentation-only change
- [ ] `@Schema` descriptions remain accurate and reflect the current (unchanged) behavior alongside the legacy notice

### Documentation & Structure
- [ ] `io.kestra.plugin.databricks.md` `## Tasks` section for `dbfs` updated with the legacy callout and cross-link to Unity Catalog Volumes
- [ ] `README.md` updated if it separately documents `dbfs` usage

---
*[View as Artifact](https://claude.ai/code/artifact/2906ee9a-e843-4c88-b29a-af8da02b825f)*

Contributor guide

No contributing guide indexed for this repository

Research direction

Wait for #250 to provide the Unity Catalog Volumes tasks and links. Start with the dbfs.Upload and dbfs.Download @Schema declarations, then inspect the ## Tasks section in io.kestra.plugin.databricks.md and README.md for separate DBFS documentation. Done means legacy wording and replacement links are present, with no behavior, annotation, or test changes.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
documentation
Issue type
Documentation
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
68/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.