Explore the possibility of generic `SqlToGcsOperator`
- Dominant language
- Python
- Stars
- 46.9k
- Forks
- 17.8k
- Avg merge
- 2d 10h
- Merged PRs (30d)
- 483
Description
### Body
Recently `SqlToS3Operator` was added in PR https://github.com/apache/airflow/pull/20807 (It also deprecated `MySqlToS3Operator`)
In Google provider Currently we have [BaseSQLToGCSOperator](https://github.com/apache/airflow/blob/1b568d73e1dfb838a3a0446e3a6063b9f27f04b8/airflow/providers/google/cloud/transfers/sql_to_gcs.py#L36) with `PrestoToGCSOperator`, `MSSQLToGCSOperator`, `MySQLToGCSOperator`, `OracleToGCSOperator`, `PostgresToGCSOperator`, `TrinoToGCSOperator`, `DummySQLToGCSOperator`
This task is to check if we can have a generic `SqlToGCSOperator` just like there is for AWS so we won't need class per each provider...
### Committer
- [X] I acknowledge that I am a maintainer/committer of the Apache Airflow project.
Contributor guide
Research direction
Start by reading airflow/providers/google/cloud/transfers/sql_to_gcs.py, especially BaseSQLToGCSOperator and its listed provider-specific subclasses. Compare this structure with the generic SqlToS3Operator introduced in PR 20807. Done means determining whether a generic SqlToGCSOperator is viable and identifying the provider classes and compatibility considerations affected by that decision.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- google-cloud, python, sql
- Domain
- cloud, data-engineering
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100