CentreForDigitalHumanities / CentreForDigitalHumanities/Textcavator

Replace required_field property for CSV corpora

Open
#1,113 0 comments 0 reactions 0 assignees View on GitHub
backend code quality
Dominant language
Python
Stars
12
Forks
3
Avg merge
3d 3m
Merged PRs (30d)
9

Description

The XML corpus supports a `required` property for corpus fields - skip the document if that field is empty.

For historic reasons, the `CSVCorpus` has a property `required_field` which refers to a column/field in the CSV file (_not_ a field in the corpus), i.e. skip all rows where that column is empty. It does not have an implementation for the `required` property for corpus fields.

This is unnecessarily confusing. It should be possible to remove `required_field` and instead use the `required` property on corpus fields. `required_field` is used in the following places:

- https://github.com/UUDigitalHumanitieslab/I-analyzer/blob/5603be785b8687ac436110fd560e06b6da60bec1/backend/corpora/parliament/germany-old.py#L31
- https://github.com/UUDigitalHumanitieslab/I-analyzer/blob/5603be785b8687ac436110fd560e06b6da60bec1/backend/corpora/parliament/canada.py#L25
- https://github.com/UUDigitalHumanitieslab/I-analyzer/blob/5603be785b8687ac436110fd560e06b6da60bec1/backend/corpora/parliament/germany-new.py#L26
- https://github.com/UUDigitalHumanitieslab/I-analyzer/blob/5603be785b8687ac436110fd560e06b6da60bec1/backend/corpora/parliament/denmark.py#L46

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by locating the CSVCorpus implementation and compare its required_field handling with the XML corpus's required corpus-field behavior. Review backend/corpora/parliament/germany-old.py, canada.py, germany-new.py, and denmark.py, then update their configurations to use required on corpus fields. Done means required_field is no longer needed and empty required corpus fields cause the corresponding document to be skipped.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data
Issue type
Refactor
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.