airbytehq / airbytehq/PyAirbyte

💡 Feature Request: Add `to_spark()` to on dataset class

Open
#173 5 comments 0 reactions 0 assignees View on GitHub
1.0 accepting pull requests good first issue spark
Dominant language
Python
Stars
344
Forks
77
Avg merge
1d 11h
Merged PRs (30d)
35

Description

There's an interesting use case here where you could use PyAirbyte directly in data pipelines that run on spark.

Currently if you want to do this, you need to do `to_pandas()` and then `spark_session.createDataFrame(issues_df, shema=my_schema)`, but this seems inefficient, plus you have to manually define the schema (for example for json blobs which are `object` in pandas but need to be `StringType` in spark, and other idiosyncrasies like pandas having 64 bit ints but spark having Int and Long).

Or maybe a spark df cache would be more efficient here?

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.