JuliaText / JuliaText/TextAnalysis.jl
TokenBuffer for preprocessing Documents
Open
- Dominant language
- Julia
- Stars
- 384
- Forks
- 92
- PR merge metrics
- No merged PRs in 30d
Description
We have been using a [**fast** TokenBuffer API](https://github.com/JuliaText/WordTokenizers.jl/blob/master/src/words/fast.jl) to speed up for various tokenizers in WordTokenizers.jl.
Referring to #141 #140, I think it might be beneficial to extend the TokenBuffer API for Documents and Corpus that TextAnalysis.jl offers (excluding NGramDocument and TokenDocument).
This can then be used to improve the performance for [preprocessing.jl](https://github.com/JuliaText/TextAnalysis.jl/blob/master/src/preprocessing.jl).
**Edit:** This could also serve as a solution for #74 #76
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.