CentreForDigitalHumanities / CentreForDigitalHumanities/Textcavator

Stable document IDs

Open
#1,157 1 comment 2 reactions 0 assignees View on GitHub
corpus
Dominant language
Python
Stars
12
Forks
3
Avg merge
3d 3m
Merged PRs (30d)
9

Description

Tags (#997) and document pages (#995) both work best when document IDs are stable between different versions of an index, i.e. do not change when the corpus is re-indexed.

This is true if you retrieve document IDs from the source data and add them as an `id` field, rather than letting elasticsearch generate IDs. This is done by some corpora but not all.

For usability, we might add an extractor function that generates unique IDs based on a set of fields that you assert as the unique descriptors of the document - which could be used if you do not have unique IDs in the source data.

Contributor guide

No contributing guide indexed for this repository

Research direction

No files or tests are identified. Start by tracing how corpora are indexed and how document IDs are currently assigned, then define the extractor's input and uniqueness guarantees; done means IDs remain stable across re-indexing and support the existing tags and document-page features.

Written by the indexing model from the issue text.

Assessment

Tech stack
elasticsearch, python
Domain
search
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.