New NLP related tables (proposal)

There are huge amounts of data being generated at hospitals every day. Up to 80% of this data is collected in an unstructured format and a large portion of it as free text. In order to extract value from this data it is necessary to turn it into structured data. This is specially relevant for clinical studies that benefit from all the data capture performed during the clinical practice. Natural language processing (NLP) solutions can structure clinical texts written by physicians, extracting and encoding relevant medical concepts and taking into account complex context such as negations, family/personal background, past events among others

While OMOP CDM is a great schema to store structured data, NLP results can get messy and complex. Although OMOP CDM v5.4 provides a note_nlp table to store these results, queries to this table can become clumsy and slow, so we designed and extended the OMOP CDM with our own NLP schema to store the results generated in the annotation process of NLP.

We propose this extension of the OMOP CDM to store the output of NLP solutions while integrating with the vocabulary normalization process of the OMOP CDM.

We presented this in the OHDSI Collaborator Showcase and were awarded a Best Community Contribution Award. You can see more details and a presentation here:

https://www.ohdsi.org/2021-global-symposium-showcase-28/

Start with the linked NLP extension documentation and schema diagram, then compare the proposal with the repository's existing OMOP CDM table definitions and DDLs. The issue does not identify files, tests, or acceptance criteria, so the required tables, integration points, and definition of done need to be agreed before implementation.