[FEA] Expose more options to nvtext char_normalizer
- Dominant language
- C++
- Stars
- 9.8k
- Forks
- 1.1k
- Avg merge
- 3d 6m
- Merged PRs (30d)
- 278
Description
**Is your feature request related to a problem? Please describe.**
nvtext char normalizer support a bunch of different normalizations:
1. Uppercase to lowercase
2. Padding ascii and unicode punctuation with spaces
3. Removes control chars
4. converts tab and newline to single space
5. strip accents
However there is limit control over which ones can be enabled vs disabled. For eg `do_lower=False` which should control lowercase also has the property of not stripping accents. (possibly to match [bertTokenizer](https://github.com/huggingface/transformers/blob/main/src/transformers/models/bert/tokenization_bert.py#L70-L72) behavior.
**Describe the solution you'd like**
Support more options such as `strip_accents`.
**Describe alternatives you've considered**
A clear and concise description of any alternative solutions or features you've considered.
**Additional context**
Add any other context, code examples, or references to existing implementations about the feature request here.
Contributor guide
Assessment
This issue has not been assessed yet.