CentreForDigitalHumanities / CentreForDigitalHumanities/Textcavator
Simplify document media
- Dominant language
- Python
- Stars
- 12
- Forks
- 3
- Avg merge
- 3d 3m
- Merged PRs (30d)
- 9
Description
Requesting image data for documents relies on python code in the corpus definition that defines how to find the relevant filename(s) for a document. In the case of some corpora, there is additional code that describes to to turn that metadata into a file attachment, which may involve merging several loose PDF pages into a single file.
This is an issue for #978
However, it should be possible to make it so that all of this complicated processing happens *before* the corpus is added to the i-analyzer interface.
The easiest case are corpora where each document is a single image file in their data directory (i.e. the ones that only have `request_media` implemented). In this case, just add a field `media` that gives this path. Of course, this field should not be indexed or shown to the user, but it can then be used to request the image from the backend, instead of calling `corpus.request_media()` each time.
For corpora with the PDF functionality, this can be combined with another step during data preparation, which is to make sure each document has a single file associated with it. This means that you execute the PDF-merging code in `corpus.get_media()` while extracting the data, so you can just save a single filename in the `media` field.
Alternatively, we could support an array of files for `media` and make this merging a general functionality instead of one that is defined separately in each corpus.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by reading the corpus implementations of request_media() and get_media(), then trace how corpus data is prepared and added to the i-analyzer interface. Compare the single-image and PDF cases with issue #978. Done means media information is prepared before interface use and supports the required document attachment behavior without repeated corpus-side processing.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- backend, data
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100