Schema for converting a PySpark Dataframe into a LeapFrame
- Dominant language
- Scala
- Stars
- 1.5k
- Forks
- 315
- PR merge metrics
- No merged PRs in 30d
Description
I'm creating a schema from a Pyspark dataframe that can be used to build a corresponding
Leapframe. My question is, how do I handle non-scalar types i.e. arrays ?
My schema looks like this.
```
df.printSchema()
root
|-- scalar_1: string (nullable = true)
|-- scalar_2: double (nullable = true)
|-- vector_1: array (nullable = true)
| |-- element: string (containsNull = true)
|-- vector_2: array (nullable = true)
| |-- element: date (containsNull = true)
```
If the Pyspark dataframe consisted of purely scalar columns, I would do the following:
```
schema = [ { "name" : field.simpleString().split(":")[0], \
"type" : field.simpleString().split(":")[1] }
for field in df.schema ]
```
Any suggestions are welcome.
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.