apache / apache/airflow

Add GCS, Azure Support for AnalyticsOperator

Open
#62,735 0 comments 0 reactions 1 assignee Claimed by @gopidesupavan View on GitHub
area:providers kind:feature provider:google provider:microsoft-azure
Dominant language
Python
Stars
46.9k
Forks
17.8k
Avg merge
2d 9h
Merged PRs (30d)
472

Description

As part of [#62232](https://github.com/apache/airflow/pull/62232), support was added for **AWS (S3)** and **LocalFileSystem** object stores when connecting from DataFusion.

However, DataFusion supports several additional object store backends that are not yet covered. According to the [DataFusion Object Store documentation](https://datafusion.apache.org/python/autoapi/datafusion/object_store/index.html), the following are also available:

- **Google Cloud Storage (GCS)**
- **Azure Blob Storage / Azure Data Lake Storage (ADLS)**

### Motivation

Extending object store support will allow users to seamlessly query data residing in a wider range of storage backends directly through the Airflow AnalyticsOperator.

### Requested Changes

- Add support for **Google Cloud Storage (GCS)** object store integration
- Add support for **Azure Blob Storage** object store integration

### Related

- PR: https://github.com/apache/airflow/pull/62232
- DataFusion Object Store API Reference: https://datafusion.apache.org/python/autoapi/datafusion/object_store/index.html

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.