apache / apache/airflow

Add service account impersonation with Google Cloud SQL Proxy in Google Cloud SQL Operators

Open
#39,546 3 comments 1 reaction 0 assignees View on GitHub
area:providers good first issue kind:feature provider:google
Dominant language
Python
Stars
46.9k
Forks
17.8k
Avg merge
2d 10h
Merged PRs (30d)
483

Description

### Description

I would like to be able to access Google Cloud SQL databases via the Google Cloud SQL Proxy and service account impersonation. This feature was introduced for some Google Cloud operators [here](https://github.com/apache/airflow/issues/8803). Currently, this is not possible since the function that manages the credentials that are passed to the Cloud SQL Proxy only handles service account key files ([relevant function](https://github.com/apache/airflow/blob/8dcee5b24d5ecfc67bdb7800ecd750d37d66be10/airflow/providers/google/cloud/hooks/cloud_sql.py#L593)) and falls back to the default Google Cloud connection when not available.

The Cloud SQL Proxy recently introduced an additional flag `--impersonate-service-account` that [adds support for service account impersonation](https://github.com/GoogleCloudPlatform/cloud-sql-proxy/issues/417).

This would require updating the cloud-sql-proxy to version 2, which would also require changes to some of the command line arguments and handling of stdout/stderr.

### Use case/motivation

We are operating a shared Google Cloud Composer environment in a single Google Cloud project, but each team is using a dedicated project for non-Airflow-related things. From the Composer service account, we delegate to project-specific service accounts via service account impersonation. This works fine for most Google Cloud Operators, but not for Cloud SQL Operators when using the Cloud SQL Proxy.

### Related issues

Another issue that might be related is the option to add IAM authentication to the operator as well: https://github.com/apache/airflow/pull/20775

### Are you willing to submit a PR?

- [ ] Yes I am willing to submit a PR!

### Code of Conduct

- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)

Contributor guide

Open the contributing guide

Research direction

Start with the credential-handling function in airflow/providers/google/cloud/hooks/cloud_sql.py around line 593, then review the Cloud SQL Proxy version 2 requirements. Trace the changed command-line arguments and stdout/stderr handling needed for service account impersonation. Done means Cloud SQL Operators can access databases through the proxy using the --impersonate-service-account option.

Written by the indexing model from the issue text.

Assessment

Tech stack
google-cloud, python
Domain
cloud, databases
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.