apache / apache/incubator-xtable
Make XTable as a community managed Airflow provider
- Dominant language
- Java
- Stars
- 1.2k
- Forks
- 212
- Avg merge
- 4d 9h
- Merged PRs (30d)
- 16
Description
### Feature Request / Improvement
Hi XTable maintainers,
I am planning to create an [Airflow operator](https://airflow.apache.org/docs/apache-airflow/stable/core-concepts/operators.html) for XTable, and also trying to make it as a [community managed Airflow provider](https://airflow.apache.org/docs/apache-airflow-providers/).
By saying Airflow operator, what I mean is something similar to what AWS presents in [this blog](https://aws.amazon.com/blogs/big-data/run-apache-xtable-on-amazon-mwaa-to-translate-open-table-formats/), which is a wrapper of XTable's java command, allowing users to trigger it with config in Python codes and as an Airflow task. I believe integrating XTable in Airflow has great benefits for making it popular, and closer to be an industry standard.
I've tried proposing this with Airflow directly, while it requires votes to add it as a provider. More importantly, they are also looking for support from XTable (or maybe even OneHouse?) directly, since they prefer "mixed governance" approach. As an example, [here](https://lists.apache.org/thread/2yrwzgzbo44fqo2rgcyqxw1sjd2sob30) is the discussion in Airflow devlist about adding a new provider. Hence, I am requesting your support to bring this discussion on the table of both sides, provide more background and evidence why XTable is helpful for data engineers (who are highly possibly Airflow users as well), and support such vote in Airflow devlist.
### Are you willing to submit PR?
- [X] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with Airflow's provider documentation and the linked Airflow devlist discussion; confirm the XTable Java command and the proposed Python operator scope with maintainers. Done would require an agreed mixed-governance path and a concrete provider contribution plan, rather than only advocacy.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, python
- Domain
- data-engineering
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100