NVIDIA / NVIDIA/cudf

[FEA] Support string column to struct column (and vice versa) conversions in `read_json`/`write_json`

Open
#19,338 0 comments 0 reactions 0 assignees View on GitHub
feature request libcudf Python
Dominant language
C++
Stars
9.8k
Forks
1.1k
Avg merge
3d 6m
Merged PRs (30d)
278

Description

**Is your feature request related to a problem? Please describe.**
https://github.com/rapidsai/cudf/pull/19307 and https://github.com/rapidsai/cudf/pull/19052 implemented cudf_polars features to convert of column of json-like strings to a struct column and vice versa. The current implementation involves converting/creating a column using a host buffer (a jsonl formatted string) when using `read_json` or `write_json`. It would be great to do this conversion entirely on device instead

1. Convert a string column with json-like strings with `read_json` to a struct column
2. Convert a struct column with `write_json` to a string column with json-like strings

**Describe the solution you'd like**
Sharing an ideal solution discussed with @wence- and @shrshi

> A better approach would be to modify the JSON tokenizer to take in the row offsets of the string column and reset the state of the automata every time we reach the last character of a row. We can process all rows using a single 'segmented' FST without the memory overhead of duplicating the contents of the string column

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.