Kotlin / Kotlin/dataframe

Unwrap top-level key-value objects in `readJson`

Open
#567 3 comments 0 reactions 1 assignee Claimed by @Jolanrensen View on GitHub
bug
Dominant language
Kotlin
Stars
1.1k
Forks
83
Avg merge
4d 12h
Merged PRs (30d)
30

Description

### Title

Unwrap top-level key-value objects in `readJson`

### Problem

Some JSON files use a top-level object as a map of records:

`{ "AFG": { ... }, "OWID_AFR": { ... }, ... }`

When reading such JSON with `keyValuePaths = listOf(JsonPath())`, DataFrame currently produces a structure with two columns:

- `key`
- `value`

where `value` remains a nested object.

To get the expected tabular shape, users have to manually call extra operations like `explode()` and `ungroup("value")`.

### Expected

`readJson` should support unwrapping top-level key-value objects into a more convenient DataFrame shape.

For example, top-level keys should become a regular column, and fields from `value` should be expanded into columns.

### Acceptance criteria

- Add support for unwrapping top-level key-value JSON objects
- Preserve the original key as a column
- Expand the nested `value` object into regular columns
- Add tests for wide JSON objects such as `{ key: object }`
- Document the behavior and recommended usage

### Motivation

Top-level key-value JSON is a common format for public datasets.

Current behavior is technically readable but inconvenient and non-obvious. Users must know the internal shape and apply extra transformations manually.

Supporting top-level unwrapping makes `readJson` produce a more useful tabular result directly and improves discoverability of JSON ingestion before 1.0.

With this data https://covid.ourworldindata.org/data/owid-covid-data.json that is surprisingly a "wide" JSON i found that keyValuePaths can be helpful. I tried this:
```
val df = DataFrame.readJson(
"https://covid.ourworldindata.org/data/owid-covid-data.json",
keyValuePaths = listOf(JsonPath())
)
```
![image](https://github.com/Kotlin/dataframe/assets/12936457/379c35da-aa48-487e-8333-8110d876669b)
![image](https://github.com/Kotlin/dataframe/assets/12936457/c9139976-a613-4c8e-8ad3-1724b02c12ff)

Much better than original that cannot even be compiled in notebooks, but for schema i want i need to `df.explode().ungroup("value")`
![image](https://github.com/Kotlin/dataframe/assets/12936457/dfc30f51-7162-4be3-a995-2db3af86d721)
![image](https://github.com/Kotlin/dataframe/assets/12936457/63852087-8f26-4d0b-a517-1629686b8c71)

@Jolanrensen is my JsonPath wrong?

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.