JuliaText / JuliaText/TextAnalysis.jl

PCRE compilation error: regular expression is too large at offset 592769

Open
#258 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Julia
Stars
384
Forks
92
PR merge metrics
No merged PRs in 30d

Description

Upon trying to remove sparse terms from a corpus via

```julia
remove_sparse_terms!(corp, .05)
```
I run into the following error message:

```julia
PCRE compilation error: regular expression is too large at offset 592769

error(::String)@error.jl:33
compile(::String, ::UInt32)@pcre.jl:128
compile(::Regex)@regex.jl:79
Regex(::String, ::UInt32, ::UInt32)@regex.jl:44
Regex@regex.jl:67[inlined]
mk_regex(::String)@preprocessing.jl:31
_combine_regex(::Set{AbstractString})@preprocessing.jl:547
_build_regex(::Languages.English, ::UInt32, ::Set{AbstractString}, ::Set{AbstractString})@preprocessing.jl:542
var"#prepare!#14"(::Set{AbstractString}, ::Set{AbstractString}, ::typeof(TextAnalysis.prepare!), ::TextAnalysis.Corpus{TextAnalysis.StringDocument{String}}, ::UInt32)@preprocessing.jl:414
remove_words!@preprocessing.jl:227[inlined]
remove_sparse_terms!(::TextAnalysis.Corpus{TextAnalysis.StringDocument{String}}, ::Float64)@preprocessing.jl:341
top-level scope@Local: 18
```

Is this a bug or might this just mean there is something wrong with one of the documents? That might be a possibility as I'm dealing with patents which can get pretty messy.

I'm on Julia 1.6.1 and TextAnalysis v0.7.3.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.