NVIDIA / NVIDIA/cudf

[FEA] Refactor logic detecting compression type when AUTO behavior is enabled

Open
#18,547 0 comments 1 reaction 0 assignees View on GitHub
cuIO feature request libcudf
Dominant language
C++
Stars
9.8k
Forks
1.1k
Avg merge
3d 6m
Merged PRs (30d)
278

Description

**Is your feature request related to a problem? Please describe.**
This issue outlines recommendations for follow-up work from @bdice and @vuule to improve AUTO behavior.

**Describe the solution you'd like**
1. Refactor the detection logic for each compression type into helper functions to simplify the fallthrough logic in the switch statement. Consider alternative approaches using std::optional (returning “maybe the detected sizes, or nullopt”) or using a std::variant holding one of the possible compression metadata values (offering more choices for what metadata might be returned by each compression type).
2. Optimize all our code paths for exiting early based on the distribution of most common compression types. Both the JSON and CSV reader detect compression for file types based on the file extension which can be error-prone. Moreover, the compression for source types like host and device buffers need to be fixed before passing on the buffers to the reader.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.