JuliaText / JuliaText/WordTokenizers.jl

`tokenize` API

Open
#11 9 comments 0 reactions 0 assignees View on GitHub
Dominant language
Julia
Stars
100
Forks
25
PR merge metrics
No merged PRs in 30d

Description

The `set_tokenizer` API seems a bit suspect here, given that it can be replaced with
```julia
const tokenize = WordTokenizers.nltk_tokenize
```
and likewise for RevTok etc, without bringing in multiple packages just to define an alias :)

I also think it's generally a good idea to expose people to higher order functions and such; people might not realise that you can just e.g. pass a custom tokenize function into a constructor rather than setting and unsetting it globally.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.