epfl-dlab / epfl-dlab/quootstrap

Extremely low recall

Open
#1 5 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
29
Forks
3
PR merge metrics
No merged PRs in 30d

Description

Thank you for your code.
I build a local copy of the code and implement my own dataLoader. The code runs smoothly. Since the Freebase is deprecated and old. I got rid of the Freebase in the code.
input: about 440m tokenized news articles from 1-2 week US English news. I did not use the spin3r data since it is too large
output: about 97000 quotations and names pairs are extracted.
But according to my own statistics on a small sample of the input about 100 articles. There are average 2 quotations in one article.
The recall rate of quootstrap is extremely lower than I expected.
Is this normal?

Some guessing:

1. Should I use the spin3r to get discoveredPatterns.txt and replace seedPattern.txt with it. Then run the code with the new seedPattern.txt against my own dataset? (This method assumes that the spin3r data contains all possible patterns that in news data. I kind of doubt this assumption)
2. I actually lowered the pattern confidences to 0 and change the M(if a pattern extracted pairs smaller than M = 5, the pattern would be discarded) to 1. The output is larger but the quality dropped drastically.
3. I also extended the iteration number to 200. But while running there are usually no more new patterns would be discovered after 3-4 iterations. So larger iterations does not help.

Feel free to point out that if I missed some critical points.

Thanks!

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.