CentreForDigitalHumanities / CentreForDigitalHumanities/Textcavator

Norwegian analysers only work on Bokmål

Open
#1,105 1 comment 0 reactions 0 assignees View on GitHub
corpus
Dominant language
Python
Stars
12
Forks
3
Avg merge
3d 3m
Merged PRs (30d)
9

Description

I hadn't given this any thought when writing the corpus definition but recently came across it: the stemmer for Norwegian is based Bokmål, the P&P Norway corpus also includes text in Nynorsk. (Different writing standards.) I assume that this is also true for the stopword list, but nltk's documentation is pretty meagre, so I can't be sure..

In the 1998-2016 corpus, about 10% percent of the speeches uses Nynorsk. (The 1814-2004 corpus does not have language metadata.) This is a pretty minor issue, so anything potential fixes would almost certainly not be worth the effort. My proposal would be to mention this in the documentation and leave it at that.

NB: I titled this "only work on" but that is a bit dramatic - to my knowledge, Bokmål and Nynorsk are not wholly incompatible.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.