Add service account impersonation with Google Cloud SQL Proxy in Google Cloud SQL Operators
- Dominant language
- Python
- Stars
- 46.9k
- Forks
- 17.8k
- Avg merge
- 2d 10h
- Merged PRs (30d)
- 483
Description
### Description
I would like to be able to access Google Cloud SQL databases via the Google Cloud SQL Proxy and service account impersonation. This feature was introduced for some Google Cloud operators [here](https://github.com/apache/airflow/issues/8803). Currently, this is not possible since the function that manages the credentials that are passed to the Cloud SQL Proxy only handles service account key files ([relevant function](https://github.com/apache/airflow/blob/8dcee5b24d5ecfc67bdb7800ecd750d37d66be10/airflow/providers/google/cloud/hooks/cloud_sql.py#L593)) and falls back to the default Google Cloud connection when not available.
The Cloud SQL Proxy recently introduced an additional flag `--impersonate-service-account` that [adds support for service account impersonation](https://github.com/GoogleCloudPlatform/cloud-sql-proxy/issues/417).
This would require updating the cloud-sql-proxy to version 2, which would also require changes to some of the command line arguments and handling of stdout/stderr.
### Use case/motivation
We are operating a shared Google Cloud Composer environment in a single Google Cloud project, but each team is using a dedicated project for non-Airflow-related things. From the Composer service account, we delegate to project-specific service accounts via service account impersonation. This works fine for most Google Cloud Operators, but not for Cloud SQL Operators when using the Cloud SQL Proxy.
### Related issues
Another issue that might be related is the option to add IAM authentication to the operator as well: https://github.com/apache/airflow/pull/20775
### Are you willing to submit a PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
Contributor guide
Research direction
Start with the credential-handling function in airflow/providers/google/cloud/hooks/cloud_sql.py around line 593, then review the Cloud SQL Proxy version 2 requirements. Trace the changed command-line arguments and stdout/stderr handling needed for service account impersonation. Done means Cloud SQL Operators can access databases through the proxy using the --impersonate-service-account option.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- google-cloud, python
- Domain
- cloud, databases
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100