NVIDIA / NVIDIA/cudf

[FEA] Faster path for calculating total output symbols in FST

Open
#17,114 4 comments 0 reactions 1 assignee Claimed by @elstehle View on GitHub
cuIO feature request libcudf
Dominant language
C++
Stars
9.8k
Forks
1.1k
Avg merge
3d 6m
Merged PRs (30d)
278

Description

**Is your feature request related to a problem? Please describe.**
To create a faster path for calculating total output symbols in FST, without redundantly computing some values, such as the state-transition vectors and write offsets in the FST.

Also, if the impact is significant, provide a way to optimize the FST kernel not to redundantly compute number of total output symbols, if it was passed with discard iterator.

**Describe the solution you'd like**
A reduce kernel to compute total number of output symbols.

**Describe alternatives you've considered**
used make_discard_iterator in https://github.com/rapidsai/cudf/pull/16978
The compiler should optimize and remove redundant parts, But still some redundant parts are still computed.
@elstehle shows it has 10% runtime impact. Impact on just FST alone would be much more.

**Additional context**
https://github.com/rapidsai/cudf/pull/16978#issuecomment-2408909217 Benchmark shows addtional 10% impact on overall JSON reader runtime.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.