CentreForDigitalHumanities / CentreForDigitalHumanities/Textcavator
Norwegian analysers only work on Bokmål
- Dominant language
- Python
- Stars
- 12
- Forks
- 3
- Avg merge
- 3d 3m
- Merged PRs (30d)
- 9
Description
I hadn't given this any thought when writing the corpus definition but recently came across it: the stemmer for Norwegian is based Bokmål, the P&P Norway corpus also includes text in Nynorsk. (Different writing standards.) I assume that this is also true for the stopword list, but nltk's documentation is pretty meagre, so I can't be sure..
In the 1998-2016 corpus, about 10% percent of the speeches uses Nynorsk. (The 1814-2004 corpus does not have language metadata.) This is a pretty minor issue, so anything potential fixes would almost certainly not be worth the effort. My proposal would be to mention this in the documentation and leave it at that.
NB: I titled this "only work on" but that is a bit dramatic - to my knowledge, Bokmål and Nynorsk are not wholly incompatible.
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.