airbytehq / airbytehq/PyAirbyte
💡 Feature Request: Add `to_spark()` to on dataset class
Open
1.0
accepting pull requests
good first issue
spark
- Dominant language
- Python
- Stars
- 344
- Forks
- 77
- Avg merge
- 1d 11h
- Merged PRs (30d)
- 35
Description
There's an interesting use case here where you could use PyAirbyte directly in data pipelines that run on spark.
Currently if you want to do this, you need to do `to_pandas()` and then `spark_session.createDataFrame(issues_df, shema=my_schema)`, but this seems inefficient, plus you have to manually define the schema (for example for json blobs which are `object` in pandas but need to be `StringType` in spark, and other idiosyncrasies like pandas having 64 bit ints but spark having Int and Long).
Or maybe a spark df cache would be more efficient here?
Contributor guide
Assessment
This issue has not been assessed yet.