CentreForDigitalHumanities / CentreForDigitalHumanities/Textcavator
Stable document IDs
- Dominant language
- Python
- Stars
- 12
- Forks
- 3
- Avg merge
- 3d 3m
- Merged PRs (30d)
- 9
Description
Tags (#997) and document pages (#995) both work best when document IDs are stable between different versions of an index, i.e. do not change when the corpus is re-indexed.
This is true if you retrieve document IDs from the source data and add them as an `id` field, rather than letting elasticsearch generate IDs. This is done by some corpora but not all.
For usability, we might add an extractor function that generates unique IDs based on a set of fields that you assert as the unique descriptors of the document - which could be used if you do not have unique IDs in the source data.
Contributor guide
No contributing guide indexed for this repository
Research direction
No files or tests are identified. Start by tracing how corpora are indexed and how document IDs are currently assigned, then define the extractor's input and uniqueness guarantees; done means IDs remain stable across re-indexing and support the existing tags and document-page features.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- elasticsearch, python
- Domain
- search
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100