CentreForDigitalHumanities / CentreForDigitalHumanities/Textcavator
Replace required_field property for CSV corpora
- Dominant language
- Python
- Stars
- 12
- Forks
- 3
- Avg merge
- 3d 3m
- Merged PRs (30d)
- 9
Description
The XML corpus supports a `required` property for corpus fields - skip the document if that field is empty.
For historic reasons, the `CSVCorpus` has a property `required_field` which refers to a column/field in the CSV file (_not_ a field in the corpus), i.e. skip all rows where that column is empty. It does not have an implementation for the `required` property for corpus fields.
This is unnecessarily confusing. It should be possible to remove `required_field` and instead use the `required` property on corpus fields. `required_field` is used in the following places:
- https://github.com/UUDigitalHumanitieslab/I-analyzer/blob/5603be785b8687ac436110fd560e06b6da60bec1/backend/corpora/parliament/germany-old.py#L31
- https://github.com/UUDigitalHumanitieslab/I-analyzer/blob/5603be785b8687ac436110fd560e06b6da60bec1/backend/corpora/parliament/canada.py#L25
- https://github.com/UUDigitalHumanitieslab/I-analyzer/blob/5603be785b8687ac436110fd560e06b6da60bec1/backend/corpora/parliament/germany-new.py#L26
- https://github.com/UUDigitalHumanitieslab/I-analyzer/blob/5603be785b8687ac436110fd560e06b6da60bec1/backend/corpora/parliament/denmark.py#L46
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by locating the CSVCorpus implementation and compare its required_field handling with the XML corpus's required corpus-field behavior. Review backend/corpora/parliament/germany-old.py, canada.py, germany-new.py, and denmark.py, then update their configurations to use required on corpus fields. Done means required_field is no longer needed and empty required corpus fields cause the corresponding document to be skipped.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data
- Issue type
- Refactor
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100