apache / apache/airflow

Databricks: sensor for jobs

Open
#21,379 4 comments 0 reactions 1 assignee Claimed by @jroachgolf84 View on GitHub
area:providers good first issue kind:feature provider:databricks
Dominant language
Python
Stars
46.9k
Forks
17.8k
Avg merge
2d 10h
Merged PRs (30d)
483

Description

### Description

A sensor which allows waiting for completion of a Databricks job or task.

### Use case/motivation

We would like to make it easy to wait for a Databricks job, similar to how [ExternalTaskSensor](https://airflow.apache.org/docs/apache-airflow/2.0.2/_api/airflow/sensors/external_task/index.html) allows waiting for other Airflow tasks. Specifically, we’d like to support waiting for:

* A specific job run to complete
* A specific task of a job run to complete

The state of job runs can be queried via the [JobRunsGet](https://docs.databricks.com/dev-tools/api/latest/jobs.html#operation/JobsRunsGet.) API.

### Related issues

_No response_

### Are you willing to submit a PR?

- [ ] Yes I am willing to submit a PR!

### Code of Conduct

- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.