CentreForDigitalHumanities / CentreForDigitalHumanities/Textcavator

Use normalisation for keyword fields

Open
#1,077 0 comments 0 reactions 0 assignees View on GitHub
backend enhancement
Dominant language
Python
Stars
12
Forks
3
Avg merge
3d 3m
Merged PRs (30d)
9

Description

An elasticsearch feature we could be using: using a [normaliser](https://www.elastic.co/guide/en/elasticsearch/reference/8.5/normalizer.html) allows you to standardise fields in the index. For example, using `lowercase` normalisation will mean that the values `Bla` and `bla` are considered identical for search, aggregations or filtering.

Advantages:
- Keyword fields become more user-friendly to search. E.g. in the parliament corpora, users can search `'pvda'` in the party field, and get documents with `'PvdA'` as the party.
- Puts less pressure on data-cleanup during source extraction.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.