NVIDIA / NVIDIA/cudf

[FEA] Add option to read JSON field as unparsed string

Open
#14,239 5 comments 0 reactions 1 assignee Claimed by @karthikeyann View on GitHub
2 - In Progress cuIO feature request libcudf Spark
Dominant language
C++
Stars
9.8k
Forks
1.1k
Avg merge
3d 6m
Merged PRs (30d)
278

Description

**Is your feature request related to a problem? Please describe.**

When reading JSON in Spark, if a field has mixed types, Spark will infer the type as String to avoid data loss due to the uncertainty of the actual data type.

For example, given this input file, Spark will read column `bar` as a numeric type and column `foo` as a string type.

```
$ cat test.json
{ "foo": [1,2,3], "bar": 123 }
{ "foo": { "a": 1 }, "bar": 456 }
```

Here is the Spark code that demonstrates this:

```
scala> val df = spark.read.json("test.json")
df: org.apache.spark.sql.DataFrame = [bar: bigint, foo: string]

scala> df.show
+---+-------+
|bar| foo|
+---+-------+
|123|[1,2,3]|
|456|{"a":1}|
+---+-------+
```

Currently, Spark RAPIDS fails for this example because cuDF does not support mixed types in a column:

```
Caused by: ai.rapids.cudf.CudfException: CUDF failure at: /home/jenkins/agent/workspace/jenkins-spark-rapids-jni_nightly-pre_release-181-cuda11/thirdparty/cudf/cpp/src/io/json/json_column.cu:577: A mix of lists and structs within the same column is not supported
at ai.rapids.cudf.Table.readJSON(Native Method)
```

**Describe the solution you'd like**
I would like the ability to specify to read certain columns as unparsed strings.

**Describe alternatives you've considered**
I am also exploring some workarounds in the Spark RAPIDS plugin.

**Additional context**

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.