[FEA] JSON reader performance projects

### Background

RAPIDS cuDF released python support for nested JSON reading in 23.02, and completed the work for Spark-RAPIDS integration in 24.12. We are aware of several improvements that cuDF could make to minimize pre- and post-processing in Spark-RAPIDS. There are also some features in the JSON reader that would benefit NeMo Curator. Finally, we have collected ideas for performance gains and improved code quality. This story issue highlights the most significant outstanding issues, and the full issue set is documented in the [Nested JSON reader](https://github.com/rapidsai/cudf/milestone/13) milestone.

### Improvements for Spark-RAPIDS
| Status | Issue | Outlook |
|--|--|--|
| | Empty lines #5712 | Currently Spark-RAPIDS checks and replaces empty lines with `{}` |
| | Add some validation options #15222 | We may need cuDF changes to allow validation of a new token type |
| 🔄 #17575 | Root-level list support | See issue in https://github.com/NVIDIA/spark-rapids/issues/11717 |

### Improvements for NeMo Curator
| Status | Issue | Outlook |
|--|--|--|
| | Add a column with filenames index in cudf.read_json #15960 | To be addressed in #17480 (confirmation needed). Difficult due to multi-source row tracking. |

### Improvements for cuDF-python
| Status | Issue | Outlook |
|--|--|--|
| | Make cuDF JSON writer default https://github.com/rapidsai/cudf/issues/16993 | |

### Performance projects
| Status | Issue | Outlook |
|--|--|--|
| | Refactoring JSON reader tree algorithms with Compressed Sparse Row (CSR) #15903 | #15979 introduced CSR data structure, and 🔄 #16205 focuses on constructing device JSON columns. Also related to #16965 |
| | Improved parsing kernel #16965 | Especially for wide/deep tables, we would benefit from processing multiple columns per kernel, instead of the column-per-kernel implementation today. Performance improvement for cases without much pruning. |
| | Faster total symbol calculation in FST #17114 | |
| | Optimizations for flat JSON | Needs scoping to estimate the potential for performance improvements |