huggingface / huggingface/datatrove

How about addding custom word_tokenizers?

Open
#254 0 comments 2 reactions 0 assignees View on GitHub
enhancement
Dominant language
Python
Stars
3.3k
Forks
302
Avg merge
2h 18m
Merged PRs (30d)
2

Description

How about addding custom word tokenizer class in `utis/word_tokenizers.py`?

the reason is following:
+ I just want not to use determined tokenizer(in `word_tokenizers.WORD_TOKENIZER_FACTORY`) but other tokenizer(such as [khaiii](https://github.com/kakao/khaiii)).
+ Some other languages can make their own tokenizer with custom tokenizer class.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with `utis/word_tokenizers.py` and inspect `word_tokenizers.WORD_TOKENIZER_FACTORY` to understand how tokenizers are selected. Consider how a custom class such as khaiii's tokenizer could be supplied and how other languages could provide their own implementations. Done should mean callers can use custom tokenizers instead of only the existing determined choices.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.