airbytehq / airbytehq/PyAirbyte

💡 Feature Request: Add `to_spark()` to on dataset class

Ouverte
#173 5 commentaires 0 réactions 0 personnes assignées Voir sur GitHub
1.0 accepting pull requests good first issue spark
Langage dominant
Python
Étoiles
344
Forks
77
Merge moyen
1 j 11 h
PR mergées (30 j)
35

Description

There's an interesting use case here where you could use PyAirbyte directly in data pipelines that run on spark.

Currently if you want to do this, you need to do `to_pandas()` and then `spark_session.createDataFrame(issues_df, shema=my_schema)`, but this seems inefficient, plus you have to manually define the schema (for example for json blobs which are `object` in pandas but need to be `StringType` in spark, and other idiosyncrasies like pandas having 64 bit ints but spark having Int and Long).

Or maybe a spark df cache would be more efficient here?

Guide de contribution

Ouvrir le guide de contribution

Évaluation

Cette issue n'a pas encore été évaluée.

Recevez les nouvelles issues par e-mail

Un résumé court des issues GitHub adaptées aux débutants.