combust / combust/mleap

Schema for converting a PySpark Dataframe into a LeapFrame

Open
#466 2 comments 0 reactions 1 assignee Claimed by @ancasarb View on GitHub
Dominant language
Scala
Stars
1.5k
Forks
315
PR merge metrics
No merged PRs in 30d

Description

I'm creating a schema from a Pyspark dataframe that can be used to build a corresponding
Leapframe. My question is, how do I handle non-scalar types i.e. arrays ?

My schema looks like this.

```
df.printSchema()
root
|-- scalar_1: string (nullable = true)
|-- scalar_2: double (nullable = true)
|-- vector_1: array (nullable = true)
| |-- element: string (containsNull = true)
|-- vector_2: array (nullable = true)
| |-- element: date (containsNull = true)
```

If the Pyspark dataframe consisted of purely scalar columns, I would do the following:

```
schema = [ { "name" : field.simpleString().split(":")[0], \
"type" : field.simpleString().split(":")[1] }
for field in df.schema ]
```
Any suggestions are welcome.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.