blaze / blaze/odo

JSON datashape discovery doesn't recognize NaN as Option type

Open
#272 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
1k
Forks
131
PR merge metrics
No merged PRs in 30d

Description

With an example CSV file with missing fields like:

```
textcol,numcol
abc,123
,456
def,
```

odo.discover correctly identifies that the columns should be Option dtypes (i.e. the columns can contain missing data):

```
>>> import odo
>>> csvfile = odo.CSV('odo_example.csv', has_header=True)
>>> odo.discover(csvfile)
dshape("var * {textcol: ?string, numcol: ?float64}")
```

When we use odo to convert this CSV to a JSON file, the JSON missing values are represented by NaN, which I don't think is entirely within the JSON spec, but the Python json package serializes it that way and I think it's reasonable. So running this code...

```
>>> jsonfile = odo.JSON('odo_example.json')
>>> odo.odo(csvfile, jsonfile)

```

...produces the following JSON data:

```
[{"numcol": 123.0, "textcol": "abc"}, {"numcol": 456.0, "textcol": NaN}, {"numcol": NaN, "textcol": "def"}]
```

However, running odo.discover on the resulting JSON file with NaN representations of missing data doesn't result in Option dtypes.

```
>>> odo.discover(jsonfile)
dshape("3 * {numcol: float64, textcol: string}")
```

I think JSON discovery should recognize NaN as missing values and set columns with NaN data as Option dtypes.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.