Pandas(python) integration w/ Apache Hudi

## JIRA info

- Link: https://issues.apache.org/jira/browse/HUDI-1407
- Type: Improvement

---

## Comments

20/Dec/20 14:58;afrinjamanbd;Hello,[~shivnarayan] This project looks interesting and I would like to work on this project.
I have good experience in java, python, and also its other python libraries like pandas, NumPy, matplotlib, Scikit. 

Can you please enlighten me with more information about how can I start? ;;;

---

01/Sep/21 15:20;josiah-b;[~afrinjamanbd]

I am pretty sure the main thought behind this ticket is to provide people with a way to use Hudi without the need to fire-up a Spark or Flink cluster.  A lot of places choose to just do their data transformations in a simple Pandas script that executes in a short-lived container on a small node in a cluster or even in some sort of cloud container service because tools like Spark or Flink are too heavy of a tool to warrant their use given the comparative cost of EMR, DataBricks, and the like.

So far, the docs I read right now only provide examples for using Hudi with Spark and/or Flink. It would be nice to be able to get all the goodies out of Hudi through simple Python script in the future that doesn't require such heavy frameworks. Pandas fits that bill since it is just a regular Python library that can be pip-installed and no additional overhead is needed to use it (unlike Spark or Flink where you need to have a configured cluster and the Spark or Flink runtime installed in order to use it).

I know I would appreciate a solution like this that enables me to avoid depending on a heavy big data tool that I don't perceive the need for.

*In this vein, however, I do have one additional opinion to provide:*

I think you would get more mileage and bang-for-the-buck out of this if, instead of integrating with Pandas, you integrated with Apache Arrow.  Arrow is a C++ project to implement efficient columnar storage of datasets in memory and provide convenient ways to read in and write out such columnar formats like Parquet. The reason I think Arrow might be a better place to start is that Arrow is being adopted by many, many languages (Python, C/C++, Go, Rust, Java, Javascript, Julia, MATLAB, R, Ruby are mentioned on the project page), so if Hudi is integrated into Arrow, it would be a smaller step, then, to provide Hudi to all of those languages, not just Python. To boot, Arrow is capable of freely converting back and forth between Arrow Table format and Pandas DataFrame format, so you get a sort of Pandas compatibility, although not direct, out of the box if you support Arrow, first.  Direct Pandas compatibility then could be kicked down the road a bit more if desired.;;;

---

08/Feb/23 12:54;charlie.briggs;It'd be fantastic to see this support added to remove the need to use Spark as an intermediate job to run generic Python workloads.

It's worth noting that:

Apache Iceberg supports reading into Apache Arrow: [https://py.iceberg.apache.org/api]
Deltalake supports reading into Apache Arrow [https://delta-io.github.io/delta-rs/python/api_reference.html#deltalake.table.DeltaTable.to_pyarrow_table]

These both also enable easy export into other engines, such as [DuckDB|https://duckdb.org/] (although Iceberg also supports this directly via the pyiceberg API).;;;

No files, tests, or entry points are identified in the issue. Start by reviewing the linked HUDI-1407 request and the existing Spark and Flink integration documentation, then determine whether the first scope should target Pandas directly or Apache Arrow. Done would require an agreed integration design, implementation scope, and validation path.

Written by the indexing model from the issue text.