apache / apache/airflow

Use Google-signed OpenID Connect in Databricks Connection

Open
#57,867 7 comments 0 reactions 0 assignees View on GitHub
area:providers kind:feature needs-triage provider:databricks
Dominant language
Python
Stars
46.9k
Forks
17.8k
Avg merge
2d 10h
Merged PRs (30d)
483

Description

### Description

Hello,
we have in Databricks mapped service accounts from GCP. The accounts can be authenticated with Google ID Tokens. Databricks Connection doesn't support this type of tokens (or I wasn't able to figure out how). It could be done similar to AAD token.

### Use case/motivation

- In Airflow, we have set GOOGLE_APPLICATION_CREDENTIALS to authenticate GCP service account A.
- In GCP and Databricks we have service account B.
- Service account A can impersonate service account B.
- Using AuthorizedSession with default credentials and https://docs.cloud.google.com/docs/authentication/get-id-token#impersonation we can get ID token for service account B at runtime.
- This token can be then used to authenticate Databricks API as service account B.

### Related issues

_No response_

### Are you willing to submit a PR?

- [x] Yes I am willing to submit a PR!

### Code of Conduct

- [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)

Contributor guide

Open the contributing guide

Research direction

Start by locating the Databricks Connection authentication entry point and reading its existing AAD token handling. Compare that flow with the Google ID token impersonation process described in the issue and determine the expected connection configuration and authentication behavior. Done means the connection can authenticate Databricks API requests with a runtime Google-signed ID token and has coverage for the new path.

Written by the indexing model from the issue text.

Assessment

Tech stack
gcp, python
Domain
backend-api-design, cloud, data-engineering
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.