[FEA] Add option to read JSON field as unparsed string
- Dominant language
- C++
- Stars
- 9.8k
- Forks
- 1.1k
- Avg merge
- 3d 6m
- Merged PRs (30d)
- 278
Description
**Is your feature request related to a problem? Please describe.**
When reading JSON in Spark, if a field has mixed types, Spark will infer the type as String to avoid data loss due to the uncertainty of the actual data type.
For example, given this input file, Spark will read column `bar` as a numeric type and column `foo` as a string type.
```
$ cat test.json
{ "foo": [1,2,3], "bar": 123 }
{ "foo": { "a": 1 }, "bar": 456 }
```
Here is the Spark code that demonstrates this:
```
scala> val df = spark.read.json("test.json")
df: org.apache.spark.sql.DataFrame = [bar: bigint, foo: string]
scala> df.show
+---+-------+
|bar| foo|
+---+-------+
|123|[1,2,3]|
|456|{"a":1}|
+---+-------+
```
Currently, Spark RAPIDS fails for this example because cuDF does not support mixed types in a column:
```
Caused by: ai.rapids.cudf.CudfException: CUDF failure at: /home/jenkins/agent/workspace/jenkins-spark-rapids-jni_nightly-pre_release-181-cuda11/thirdparty/cudf/cpp/src/io/json/json_column.cu:577: A mix of lists and structs within the same column is not supported
at ai.rapids.cudf.Table.readJSON(Native Method)
```
**Describe the solution you'd like**
I would like the ability to specify to read certain columns as unparsed strings.
**Describe alternatives you've considered**
I am also exploring some workarounds in the Spark RAPIDS plugin.
**Additional context**
Contributor guide
Assessment
This issue has not been assessed yet.