apache / apache/airflow

Redesign Databricks Workflow Repair Functionality for Airflow 3 Compatibility

Open
#52,280 2 comments 0 reactions 0 assignees View on GitHub
area:API kind:feature needs-triage provider:databricks
Dominant language
Python
Stars
46.9k
Forks
17.8k
Avg merge
2d 10h
Merged PRs (30d)
483

Description

### Description

Following the enhancements in [PR #52228](https://github.com/apache/airflow/pull/52228), which aligns the Databricks workflow provider plugin to support Airflow 3 and introduces XCom-based storage for extra links, this issue tracks the remaining work to reintroduce and redesign the repair functionality for compatibility with Airflow 3.

Background:
Airflow 3 blocks constructs like @provide_session and direct database access, moving toward a secure and API-driven model. The current repair flow is incompatible with this model for the following reasons:

1. Extra links (e.g., “See Databricks Job Run”) are now stored in XCom at task execution time. However, Databricks’ repair API requires knowing which specific tasks to repair at the time of invocation, which is only possible after failures occur — creating a timing mismatch.
2. The current repair implementation relies on a Flask AppBuilder (FAB) view exposed via the `/repair_databricks_job endpoint`, which does not align with the future direction of Airflow extensibility using FastAPI based endpoint.

Goals:
1. Re-evaluate the approach to repair in Airflow 3, given the new architecture for link computation and the runtime ambiguity around which tasks will need repair.
2. Explore how the repair functionality can be implemented as a FastAPI-based endpoint, aligning with Airflow’s evolving API architecture.
3. Investigate authentication and authorization patterns for securing a FastAPI repair endpoint.
4. Consider deferring the repair redesign until after the plugin extensibility improvements planned for Airflow 3.1 are available, to ensure better alignment with long-term architecture. This may offer a cleaner and more sustainable path for integrating custom endpoints and plugin-level APIs.

I’d be glad if someone from the community is interested in picking this up or would like to collaborate with me on it — especially if they’re familiar with the topic and Airflow 3.

### Use case/motivation

_No response_

### Related issues

related: #40587

### Are you willing to submit a PR?

- [ ] Yes I am willing to submit a PR!

### Code of Conduct

- [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)

Contributor guide

Open the contributing guide

Research direction

Start by reading PR #52228 and the existing `/repair_databricks_job` flow, then review related issue #40587. Compare the current FAB-based approach with Airflow 3’s FastAPI extensibility and authentication patterns. Done means an agreed repair design that resolves task-selection timing and defines the endpoint’s security and compatibility path.

Written by the indexing model from the issue text.

Assessment

Tech stack
fastapi, python
Domain
api, authentication, authorization, backend
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.