[FEA] support '\n', '\r' and '\r\n' atthe same time as line delimiters for CSV parsing
- Dominant language
- C++
- Stars
- 9.8k
- Forks
- 1.1k
- Avg merge
- 3d 6m
- Merged PRs (30d)
- 278
Description
**Is your feature request related to a problem? Please describe.**
The default setting for Spark when reading CSV for line delimiters is '\r' (Carriage Return), '\n' (Line Feed), and/or '\r\n' (Carriage Return followed by Line Feed)
Currently in the Spark plugin we pre-process the CSV input data before sending it to CUDF for parsing. The pre-processing handles splits to match what Spark currently does and also fixing the line delimiters to be a single uniform value. We have found that with fast storage replacing the line delimiters is a real bottleneck.
We are also concerned about being ready to support GPU Direct Storage where we would not be able to pre-process the data before sending it to cudf.
**Describe the solution you'd like**
We would like an option when parsing CSV to have CUDF recognize '\r' (Carriage Return), '\n' (Line Feed), and '\r\n' (Carriage Return followed by Line Feed) all as valid line delimiters at the same time.
**Describe alternatives you've considered**
Keep doing what we are doing and be slower than ideal when parsing CSV and not be able to support CSV without config modifications when we do adopt GU Direct Storage.
Contributor guide
Assessment
This issue has not been assessed yet.