JuliaText / JuliaText/TextAnalysis.jl

Restricting dtm/tf_idf creation to only the top N features from the lexicon

Open
#71 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Julia
Stars
384
Forks
92
PR merge metrics
No merged PRs in 30d

Description

I am working with a corpus of 100k+ documents, so the number of features in the lexicon is extremely high. Thus I'm running into memory issues and the like. I know in scipy's TfidfVectorizer and similar approaches, you can limit the number of features such that you only are dealing with the top N features when working with the dtm and tf_idf matrices. Is there some way to do that with this package, or are there plans to add such a feature?

Thanks!

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.