h2oai / h2oai/datatable

Implement JSON files parsing in fread

Open
#877 0 comments 0 reactions 0 assignees View on GitHub
fread improve low priority
Dominant language
C++
Stars
1.9k
Forks
164
Avg merge
7h 31m
Merged PRs (30d)
1

Description

JSON files have a structure similar to the following:
```
{"key1": value1, "key2": value2, "key3": value3, ...},
{"key1": value1, "key2": value2, "key3": value3, ...},
{"key2": value1, "key3": value2, "key1": value3, ...},
...
```
We expect each record to have the same set of fields, but possibly in different order (owing to the fact that JSON writer probably had unordered dictionary for each record).

A complication arises if JSON has non-trivial nested structure. That is, one of the values in each record is either array or an object. For such cases, the following solution can be adopted:
* if value is an array of objects (e.g. `[{...}, {...}, ...]`) then create a new Frame (in addition to the "main" frame returned) containing "fieldN.key" and as many columns as there are keys in each object, while the original Frame will contain only "fieldN.key". Then when reading each record, we'll store a single key in the main Frame, and add multiple records into the "fieldN" Frame.
* if value is an array of primitives (e.g. `[12, -6, 5, ...]`), then treat as if it was an array of single-record objects: `[{"value":12}, {"value":-6}, {"value":5}]`.
* if value is an array of arrays, then again treat it as an array of single-value objects, where each value is an array: `[[1, 2, 3], [4], [5, 6]]` becomes `[{"array": [1, 2, 3]}, {"array": [4]}, {"array": [5, 6]}]`. Thus, array-of-arrays creates 2 auxiliary Frames.
* if value is an object, then we assume those objects may have different set of keys for different top-level records. Thus, we treat `{"subkey1": val1, "subkey2": val2, ...}` as `[{"key": "subkey1", "value": val1}, {"key": "subkey2", "value": val2}, ...]`.

This decomposition must be taken recursively.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.