Case Study: Large scale text classification
- Dominant language
- Python
- Stars
- 951
- Forks
- 262
- PR merge metrics
- No merged PRs in 30d
Description
It would be useful to build a case study for large scale text classification. This would presumably exercise preprocessing with hashing vectorizers. It would also be interesting compare newer systems like Keras-based neural-nets against traditional bag-of-bigrams and logistic regression.
A list of possibly interesting datasets can be found in [this paper, page 5](https://papers.nips.cc/paper/5782-character-level-convolutional-networks-for-text-classification.pdf), though presumably others are around as well.
```
Table 3: Statistics of our large-scale datasets. Epoch size is the number of minibatches in one epoch
Dataset Classes Train Samples Test Samples Epoch Size
AG’s News 4 120,000 7,600 5,000
Sogou News 5 450,000 60,000 5,000
DBPedia 14 560,000 70,000 5,000
Yelp Review Polarity 2 560,000 38,000 5,000
Yelp Review Full 5 650,000 50,000 5,000
Yahoo! Answers 10 1,400,000 60,000 10,000
Amazon Review Full 5 3,000,000 650,000 30,000
Amazon Review Polarity 2 3,600,000 400,000 30,000
```
This came out of conversation with @ogrisel
Contributor guide
Assessment
This issue has not been assessed yet.