Implement JSON files parsing in fread
- Dominant language
- C++
- Stars
- 1.9k
- Forks
- 164
- Avg merge
- 7h 31m
- Merged PRs (30d)
- 1
Description
JSON files have a structure similar to the following:
```
{"key1": value1, "key2": value2, "key3": value3, ...},
{"key1": value1, "key2": value2, "key3": value3, ...},
{"key2": value1, "key3": value2, "key1": value3, ...},
...
```
We expect each record to have the same set of fields, but possibly in different order (owing to the fact that JSON writer probably had unordered dictionary for each record).
A complication arises if JSON has non-trivial nested structure. That is, one of the values in each record is either array or an object. For such cases, the following solution can be adopted:
* if value is an array of objects (e.g. `[{...}, {...}, ...]`) then create a new Frame (in addition to the "main" frame returned) containing "fieldN.key" and as many columns as there are keys in each object, while the original Frame will contain only "fieldN.key". Then when reading each record, we'll store a single key in the main Frame, and add multiple records into the "fieldN" Frame.
* if value is an array of primitives (e.g. `[12, -6, 5, ...]`), then treat as if it was an array of single-record objects: `[{"value":12}, {"value":-6}, {"value":5}]`.
* if value is an array of arrays, then again treat it as an array of single-value objects, where each value is an array: `[[1, 2, 3], [4], [5, 6]]` becomes `[{"array": [1, 2, 3]}, {"array": [4]}, {"array": [5, 6]}]`. Thus, array-of-arrays creates 2 auxiliary Frames.
* if value is an object, then we assume those objects may have different set of keys for different top-level records. Thus, we treat `{"subkey1": val1, "subkey2": val2, ...}` as `[{"key": "subkey1", "value": val1}, {"key": "subkey2", "value": val2}, ...]`.
This decomposition must be taken recursively.
Contributor guide
Assessment
This issue has not been assessed yet.