apache / apache/airflow

Proposal: Add MariaDB Provider with ColumnStore and S3 Support

Open
#56,846 2 comments 0 reactions 0 assignees View on GitHub
area:providers kind:feature needs-triage
Dominant language
Python
Stars
46.9k
Forks
17.8k
Avg merge
2d 10h
Merged PRs (30d)
483

Description

### Description

## Proposal: MariaDB Provider for Apache Airflow

### Overview
I would like to contribute a comprehensive MariaDB provider to Apache Airflow that includes:
- Native MariaDB connector support
- ColumnStore engine integration
- Bulk data loading with cpimport utility
- S3 integration for data transfer
- SSH-based remote execution

### Why a Separate MariaDB Provider?
While MySQL provider exists, MariaDB has unique features that justify a dedicated provider:
1. **Native MariaDB Connector**: Uses `mariadb` library instead of generic MySQL connector
2. It is seen that using Native Mariadb as a python connector performs 2-3x better for operations like insert_many, json handling and select queries.
3. **cpimport Utility**: Bulk loading functionality specific to MariaDB ColumnStore
4. **ColumnStore Engine Support**: Validation and optimization for ColumnStore tables
5. **S3 Integration**: Direct S3 load/dump operations
6. **SSH Integration**: Remote execution capabilities
### Components Included
- **MariaDBHook**: Main hook with native connector support
- **MariaDBOperator**: SQL execution operator
- **MariaDBCpImportOperator**: Bulk data loading for ColumnStore
- **MariaDBS3LoadOperator**: Load data from S3 to MariaDB
- **MariaDBS3DumpOperator**: Export MariaDB data to S3
- **Comprehensive test suite** with 95%+ coverage
- **Example DAGs** for all features
- **Complete documentation**

### Dependencies
- `mariadb>=1.1.0` (native connector)
- `boto3>=1.26.0` (S3 integration)
- `paramiko>=2.7.0` (SSH support)

### Implementation Status
✅ Complete provider implementation
✅ Comprehensive test suite
✅ Example DAGs
✅ Documentation
✅ Apache License headers
✅ Type hints throughout

### Questions for Community
1. Is there interest in a dedicated MariaDB provider?
2. Are there any specific requirements or concerns?
3. Should I proceed with creating a pull request?

### Use case/motivation

_No response_

### Related issues

_No response_

### Are you willing to submit a PR?

- [x] Yes I am willing to submit a PR!

### Code of Conduct

- [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)

Contributor guide

Open the contributing guide

Research direction

Start by reviewing Airflow's existing MySQL provider and comparing it with the proposed MariaDBHook, MariaDBOperator, cpimport and S3 operators. Confirm the community's requirements and scope before work begins; done means the agreed provider components, tests, example DAGs, dependencies, and documentation are complete.

Written by the indexing model from the issue text.

Assessment

Tech stack
mariadb, python
Domain
cloud, data-engineering, databases
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.