CentreForDigitalHumanities / CentreForDigitalHumanities/langpro-annotator
Pre-parse sentences and save in DB
- Dominant language
- TypeScript
- Stars
- 0
- Forks
- 0
- Avg merge
- 5d 11h
- Merged PRs (30d)
- 2
Description
Currently, LangPro will both parse and prove each problem it receives. Especially the parsing takes a lot of time, so we stand to gain a lot of time by preparsing every sentence in the corpus and storing the raw Prolog results for the CCG Tree (the other parse trees can be derived very quickly) in the DB.
When the user hits 'Parse and Proof' in the annotator, the backend will first retrieve the parse results from the DB (if they are available) and send them to LangPro, which will simply have to derive the other three parse trees and do the proving.
This requires
- a new endpoint on LangPro Container (e.g. `/api/parse`) that takes a problem as its input and returns just the parse output.
- a new endpoint on LangPro Container or an update to the existing `/api/prove` route: it should be able to receive parse results and simply run the prover (bypassing the parser). If we implement a new endpoint, the output should have the same format as `/api/prove`.
- an asynchronous worker that can be triggered to go over all problems, send them to LangPro one by one to be parsed and then store the results in the DB.
- DB models for the stored parse results.
For the async worker, we can use a Celery/RabbitMQ setup. If we're comfortable upgrading to Django 6.0, there is also the new ['Tasks'](https://docs.djangoproject.com/en/6.0/topics/tasks/) module to consider.
Contributor guide
Assessment
This issue has not been assessed yet.