JuliaText / JuliaText/WordTokenizers.jl
`tokenize` API
Open
- Dominant language
- Julia
- Stars
- 100
- Forks
- 25
- PR merge metrics
- No merged PRs in 30d
Description
The `set_tokenizer` API seems a bit suspect here, given that it can be replaced with
```julia
const tokenize = WordTokenizers.nltk_tokenize
```
and likewise for RevTok etc, without bringing in multiple packages just to define an alias :)
I also think it's generally a good idea to expose people to higher order functions and such; people might not realise that you can just e.g. pass a custom tokenize function into a constructor rather than setting and unsetting it globally.
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.