Glavin001 / Glavin001/IssueBot

Detect which classes are difficult to predict

Open
#14 0 comments 0 reactions 1 assignee Claimed by @Glavin001 View on GitHub
enhancement
Dominant language
JavaScript
Stars
14
Forks
5
PR merge metrics
No merged PRs in 30d

Description

If you look at the scores you can see that certain classes are very difficult to predict and should be ignored.
### Example

For example, take https://github.com/Glavin001/atom-beautify
#### Before

`We correctly predicted 573 of 835 labelled issues, obtaining a score of 68.62275449101797%!`

```
precision recall f1-score support

add-beautifier 0.97 1.00 0.99 37
add-beautifier-option 1.00 0.93 0.97 30
add-language 1.00 1.00 1.00 144
bug 1.00 0.94 0.97 103
deprecated-atom-api 0.93 0.74 0.82 19
enhancement 1.00 1.00 1.00 134
feature 1.00 1.00 1.00 58
high priority 1.00 0.32 0.48 95
in-progress 1.00 0.36 0.53 11
invalid 1.00 0.56 0.72 66
mac 1.00 0.50 0.67 4
published 1.00 0.15 0.26 164
question 1.00 0.93 0.97 168
quick-todo 0.00 0.00 0.00 12
third-party bug 0.96 0.99 0.98 139
update third-party dependency 1.00 0.31 0.47 13
user-update-atom 0.00 0.00 0.00 4
waiting-for-user-information 0.81 0.53 0.64 32
windows 0.93 1.00 0.96 25
wontfix 0.00 0.00 0.00 7

avg / total 0.97 0.75 0.80 1265
```
#### After

Ignore the following labels to achieve 97%

``` python
ignore_labels = ['quick-todo', 'wontfix', 'user-update-atom', 'in-progress', 'pending-publication', 'published', 'waiting-for-user-information', 'high priority']
```

```
precision recall f1-score support

add-beautifier 0.97 1.00 0.99 37
add-beautifier-option 1.00 0.93 0.97 30
add-language 1.00 1.00 1.00 144
bug 1.00 0.97 0.99 103
deprecated-atom-api 1.00 0.84 0.91 19
enhancement 1.00 1.00 1.00 134
feature 1.00 1.00 1.00 58
invalid 1.00 0.53 0.69 66
mac 1.00 0.50 0.67 4
question 1.00 0.95 0.98 168
third-party bug 0.97 0.99 0.98 139
update third-party dependency 1.00 0.31 0.47 13
windows 0.93 1.00 0.96 25

avg / total 0.99 0.94 0.96 940
```

---

Resources:
- http://stats.stackexchange.com/questions/117654/what-does-the-numbers-in-the-classification-report-of-sklearn-mean
- http://scikit-learn.org/stable/modules/generated/sklearn.metrics.precision_recall_fscore_support.html

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.