dselivanov / dselivanov/text2vec
Topic modeling guide
- Dominant language
- R
- Stars
- 876
- Forks
- 133
- PR merge metrics
- No merged PRs in 30d
Description
It will be useful to create a comprehensive practical guide for topic modeling. Now we have all components in place:
- POS tags and lemmatization - thanks to `udpipe` package
- `coherence` measures - thanks to Manuel work
- fast LDA, thanks to WarpLDA in text2vec
- fast non-negative matrix factorization, thanks to `rsparse` package
- multi-word phrase extraction - several approaches `text2vec::Collocations`, `udpipe::as_phrasemachine`
### Steps
- [ ] find interesting non-trivial corpus with large number of documents
- [ ] demonstrate how to create tokenizer which only use particular POS
- [ ] create collocation model on top of that
- [ ] create document-term matrix using tokens with multi-word expression
- [ ] fit several topic models (`text2vec::LDA`, `rsparse::WRMF`) with different hyper parameters
- [ ] cross-validate / compare them using different coherence metrics
- [ ] demonstrate usage of external corpus for `tcm` calculation
- [ ] check on how coherence metrics are correlated (is perplexity correlated with them? )
There are already good vignettes in udpipe package [topic modeling](https://cran.r-project.org/web/packages/udpipe/vignettes/udpipe-usecase-topicmodelling.html) and [phrase extraction](https://cran.r-project.org/web/packages/udpipe/vignettes/udpipe-usecase-postagging-lemmatisation.html). They can be used as inspiration.
@manuelbickel @jwijffels anything we can add to the plan above?
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.