huggingface / huggingface/datatrove
How about addding custom word_tokenizers?
- Dominant language
- Python
- Stars
- 3.3k
- Forks
- 302
- Avg merge
- 2h 18m
- Merged PRs (30d)
- 2
Description
How about addding custom word tokenizer class in `utis/word_tokenizers.py`?
the reason is following:
+ I just want not to use determined tokenizer(in `word_tokenizers.WORD_TOKENIZER_FACTORY`) but other tokenizer(such as [khaiii](https://github.com/kakao/khaiii)).
+ Some other languages can make their own tokenizer with custom tokenizer class.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with `utis/word_tokenizers.py` and inspect `word_tokenizers.WORD_TOKENIZER_FACTORY` to understand how tokenizers are selected. Consider how a custom class such as khaiii's tokenizer could be supplied and how other languages could provide their own implementations. Done should mean callers can use custom tokenizers instead of only the existing determined choices.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100