INCATools / INCATools/ontology-access-kit
Add solr backend
- Dominant language
- Python
- Stars
- 198
- Forks
- 35
- Avg merge
- 3d 4h
- Merged PRs (30d)
- 1
Description
This would live here:
https://github.com/INCATools/ontology-access-kit/tree/main/src/oaklib/implementations/solr
(note: the folder named 'solor' is not a typo. It means this: http://solor.io/)
This should support both read and write. A main use case is making a solr index for an ontology.
TBD:
1. decide on a single datamodel everyone uses
2. make this completely flexible
3. some hybrid
I think 3 is best, with a core set of standardized fields corresponding to Basic, OboGraph, OMO. But people will likely want custom closures and custom annotation properties.
I am envisioning two or three document types, in order of priority/importance
1. node/entity
2. edge/logical axiom
3. annotation assertion
node would have fields like:
- id
- uri
- label
- owl type or other metatype [class, OP, AP, DP, ontology, subset, named individual, anonymous individual]
- categories
- one field per language label
- aliases (flat list)
- structured aliases (includes predicate type, annotations, lang tag, ...)
- obsoletion status
- replaced by
- definition
- structured definition
- mappings, flat list
- structured mappings, in sssom tsv syntax
- structured mappings, in sssom json
- one field per omo property used
- structured annotations/metadata
- direct is_a
- is_a ancestors
- structured outgoing relationships, direct
- is_a_closure
- is_a_part_of_closure
- optional additional closures
- structured ancestors (basically tuples of predicate-object from relation graph)
- direct is_a children
- structured incoming relationships, direct
- would NOT include descendants, this would explode
- history blob, as kgcl
- obo json blob
- rdf/xml blob
- owl functional syntax blob
(refer to the OAK glossary for the meanings of these terms in the context of OAK, and most of these correspond to method calls in various oak interfaces)
the edge/logical axiom table would have
- subject_id
- subject_label
- predicate_id
- predicate_label
- object_id
- object_label
with obvious semantics. These would be generally always populated
we would also include closures, e.g
- {sub,ob}ject is-a closure
- {sub,ob}ject is-a-part-of closure
- {sub,ob}ject optional closures
and additional fields of axiom annotation
- sources
- structured axiom annotations (rare, as we rarely have 2nd order annotation)
- sssom fields
- ...
The 3rd doc type for annotation assertions would largely follow this but would allow literals, following rdftab
- object_value
- object_lang
- object_datatype
Note that although this issue is about solr access, this can be seen as a composition:
1. normalizer (e.g. robot template)/denormalizer (e.g. simple flat file reports)
2. closurizer (i.e basic joins with relation-graph results)
3. ismorphic loading of a TSV-like model into Solr, or any other kind of denormalized store (ES, bigtable, column store DBs, ...)
See my document [Semantic Column-oriented Data Store Patterns](https://docs.google.com/document/d/1GoTZd4HSHI9q48Q6WUR4eDgBy0WgwsXcfVBihs_l0CU/edit)
Contributor guide
Research direction
Start by reviewing src/oaklib/implementations/solr and the OAK glossary to understand the proposed backend location and terminology. The issue leaves the data model, document types, flexibility, normalization, closure handling, and completion criteria undecided, so a contributor would need those decisions before implementation can be considered done.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- backend, databases
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 15/100