NVIDIA / NVIDIA/cudf

[FEA] Expose more options to nvtext char_normalizer

Open
#23,443 2 comments 0 reactions 1 assignee Claimed by @davidwendt View on GitHub
feature request
Dominant language
C++
Stars
9.8k
Forks
1.1k
Avg merge
3d 6m
Merged PRs (30d)
278

Description

**Is your feature request related to a problem? Please describe.**

nvtext char normalizer support a bunch of different normalizations:
1. Uppercase to lowercase
2. Padding ascii and unicode punctuation with spaces
3. Removes control chars
4. converts tab and newline to single space
5. strip accents

However there is limit control over which ones can be enabled vs disabled. For eg `do_lower=False` which should control lowercase also has the property of not stripping accents. (possibly to match [bertTokenizer](https://github.com/huggingface/transformers/blob/main/src/transformers/models/bert/tokenization_bert.py#L70-L72) behavior.

**Describe the solution you'd like**
Support more options such as `strip_accents`.

**Describe alternatives you've considered**
A clear and concise description of any alternative solutions or features you've considered.

**Additional context**
Add any other context, code examples, or references to existing implementations about the feature request here.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.