[FEA] Add a column with filenames index in cudf.read_json
- Dominant language
- C++
- Stars
- 9.8k
- Forks
- 1.1k
- Avg merge
- 3d 6m
- Merged PRs (30d)
- 278
Description
Hi!
`cudf.read_json` supports passing multiple files to it, which is much more performant than reading json files individually, and then merging them.
It would be very useful for certain workloads to add a column containing the index to the files passed to cudf.read_json method that indicates to which file corresponds each row in the dataset.
I would suggest adding a new input parameter, named something similar to `input_file_indexes_series_name` with a default value of None, and, when populated with a string, it would indicate that the indexes of the input files passed to `cudf.read_json` should be added to a column named as detailed in `input_file_indexes_series_name` parameter.
Already discussed with @vuule.
Thanks!
P.S.: NeMo Curator may benefit of this FR when performing Exact Deduplication, Fuzzy Deduplication and Download and Extract corpus features.
P.S.: This FR was originally raised [here](https://github.com/rapidsai/cudf/issues/15559).
Contributor guide
Assessment
This issue has not been assessed yet.