NVIDIA / NVIDIA/cudf

[FEA] Add a column with filenames index in cudf.read_json

Open
#15,960 6 comments 0 reactions 0 assignees View on GitHub
feature request Python
Dominant language
C++
Stars
9.8k
Forks
1.1k
Avg merge
3d 6m
Merged PRs (30d)
278

Description

Hi!

`cudf.read_json` supports passing multiple files to it, which is much more performant than reading json files individually, and then merging them.

It would be very useful for certain workloads to add a column containing the index to the files passed to cudf.read_json method that indicates to which file corresponds each row in the dataset.

I would suggest adding a new input parameter, named something similar to `input_file_indexes_series_name` with a default value of None, and, when populated with a string, it would indicate that the indexes of the input files passed to `cudf.read_json` should be added to a column named as detailed in `input_file_indexes_series_name` parameter.

Already discussed with @vuule.

Thanks!

P.S.: NeMo Curator may benefit of this FR when performing Exact Deduplication, Fuzzy Deduplication and Download and Extract corpus features.
P.S.: This FR was originally raised [here](https://github.com/rapidsai/cudf/issues/15559).

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.