apache / apache/incubator-xtable

Make XTable as a community managed Airflow provider

Open
#495 8 comments 3 reactions 0 assignees View on GitHub
enhancement
Dominant language
Java
Stars
1.2k
Forks
212
Avg merge
4d 9h
Merged PRs (30d)
16

Description

### Feature Request / Improvement

Hi XTable maintainers,

I am planning to create an [Airflow operator](https://airflow.apache.org/docs/apache-airflow/stable/core-concepts/operators.html) for XTable, and also trying to make it as a [community managed Airflow provider](https://airflow.apache.org/docs/apache-airflow-providers/).

By saying Airflow operator, what I mean is something similar to what AWS presents in [this blog](https://aws.amazon.com/blogs/big-data/run-apache-xtable-on-amazon-mwaa-to-translate-open-table-formats/), which is a wrapper of XTable's java command, allowing users to trigger it with config in Python codes and as an Airflow task. I believe integrating XTable in Airflow has great benefits for making it popular, and closer to be an industry standard.

I've tried proposing this with Airflow directly, while it requires votes to add it as a provider. More importantly, they are also looking for support from XTable (or maybe even OneHouse?) directly, since they prefer "mixed governance" approach. As an example, [here](https://lists.apache.org/thread/2yrwzgzbo44fqo2rgcyqxw1sjd2sob30) is the discussion in Airflow devlist about adding a new provider. Hence, I am requesting your support to bring this discussion on the table of both sides, provide more background and evidence why XTable is helpful for data engineers (who are highly possibly Airflow users as well), and support such vote in Airflow devlist.

### Are you willing to submit PR?

- [X] Yes I am willing to submit a PR!

### Code of Conduct

- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with Airflow's provider documentation and the linked Airflow devlist discussion; confirm the XTable Java command and the proposed Python operator scope with maintainers. Done would require an agreed mixed-governance path and a concrete provider contribution plan, rather than only advocacy.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, python
Domain
data-engineering
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.