CentreForDigitalHumanities / CentreForDigitalHumanities/Textcavator

Add lemmatized data to ES index

Open
#1,506 1 comment 0 reactions 0 assignees View on GitHub
backend enhancement frontend
Dominant language
Python
Stars
12
Forks
3
Avg merge
3d 3m
Merged PRs (30d)
9

Description

**Is your feature request related to a problem? Please describe.**
For the word models of Finnish parliamentary data, the preprocessing included lemmatization with spaCy's `fi_core_news_lg` model. Researchers are now wondering whether we could also have the lemmatized data available in Elasticsearch.

**Describe the solution you'd like**
We can arguably add another subfield of `speech` and add lemmatized data. Unfortunately, the lemmatized data from the word embedding preprocessing cannot be reused, as this chunks all speeches from one year together.

**Alternative solutions**
Leave analysis of lemmatized data out of the scope of I-Analyzer - we could also suggest that users need to roll their own.

**Problems**
We would not be able to apply lemmatization to the search query as well.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.