airbytehq / airbytehq/PyAirbyte
💡 Feature Request: Add `to_spark()` to on dataset class
Ouverte
1.0
accepting pull requests
good first issue
spark
- Langage dominant
- Python
- Étoiles
- 344
- Forks
- 77
- Merge moyen
- 1 j 11 h
- PR mergées (30 j)
- 35
Description
There's an interesting use case here where you could use PyAirbyte directly in data pipelines that run on spark.
Currently if you want to do this, you need to do `to_pandas()` and then `spark_session.createDataFrame(issues_df, shema=my_schema)`, but this seems inefficient, plus you have to manually define the schema (for example for json blobs which are `object` in pandas but need to be `StringType` in spark, and other idiosyncrasies like pandas having 64 bit ints but spark having Int and Long).
Or maybe a spark df cache would be more efficient here?
Guide de contribution
Ouvrir le guide de contribution
Évaluation
Cette issue n'a pas encore été évaluée.