NVIDIA / NVIDIA/cudf

JSON spark reader plan for 24.12

Open
#17,138 0 comments 1 reaction 0 assignees View on GitHub
cuIO improvement Spark
Dominant language
C++
Stars
9.8k
Forks
1.1k
Avg merge
3d 6m
Merged PRs (30d)
278

Description

These are the planned optimizations and bug fixes for JSON spark reader for 24.12 release.

- Memory optimization PR https://github.com/rapidsai/cudf/pull/16978
- Runtime mitigation issue - Multi-stage FST implementation (Elias, Shruti) https://github.com/rapidsai/cudf/issues/17114
- input schema issue/PR (New Feature)
- https://github.com/rapidsai/cudf/issues/17091,
- https://github.com/rapidsai/cudf/issues/17090,
- https://github.com/rapidsai/cudf/issues/17002,
- https://github.com/rapidsai/cudf/issues/16799,
- https://github.com/rapidsai/cudf/issues/16797
- PR https://github.com/rapidsai/cudf/pull/17029
- Performance: Preprocessing: nullify empty lines PR https://github.com/rapidsai/cudf/pull/17028
- Bugfix: last invalid json is not error - https://github.com/rapidsai/cudf/issues/16999
- Bugfix: disable array of arrays for spark - https://github.com/rapidsai/cudf/pull/17030
- Performance: mega kernel - https://github.com/rapidsai/cudf/issues/16965

- https://github.com/NVIDIA/spark-rapids/issues/11560 (input schema, and post-processing move columns without copying)
- https://github.com/NVIDIA/spark-rapids-jni/pull/2510

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.