JSON datashape discovery doesn't recognize NaN as Option type
- Dominant language
- Python
- Stars
- 1k
- Forks
- 131
- PR merge metrics
- No merged PRs in 30d
Description
With an example CSV file with missing fields like:
```
textcol,numcol
abc,123
,456
def,
```
odo.discover correctly identifies that the columns should be Option dtypes (i.e. the columns can contain missing data):
```
>>> import odo
>>> csvfile = odo.CSV('odo_example.csv', has_header=True)
>>> odo.discover(csvfile)
dshape("var * {textcol: ?string, numcol: ?float64}")
```
When we use odo to convert this CSV to a JSON file, the JSON missing values are represented by NaN, which I don't think is entirely within the JSON spec, but the Python json package serializes it that way and I think it's reasonable. So running this code...
```
>>> jsonfile = odo.JSON('odo_example.json')
>>> odo.odo(csvfile, jsonfile)
```
...produces the following JSON data:
```
[{"numcol": 123.0, "textcol": "abc"}, {"numcol": 456.0, "textcol": NaN}, {"numcol": NaN, "textcol": "def"}]
```
However, running odo.discover on the resulting JSON file with NaN representations of missing data doesn't result in Option dtypes.
```
>>> odo.discover(jsonfile)
dshape("3 * {numcol: float64, textcol: string}")
```
I think JSON discovery should recognize NaN as missing values and set columns with NaN data as Option dtypes.
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.