INCATools / INCATools/ontology-access-kit

FHIR dumper updates

Open
#369 2 comments 0 reactions 1 assignee Claimed by @joeflack4 View on GitHub
enhancement
Dominant language
Python
Stars
198
Forks
35
Avg merge
3d 4h
Merged PRs (30d)
1

Description

# Overview
Followed from:
- #367

We have a preliminary dumper: https://github.com/INCATools/ontology-access-kit/releases/tag/v0.1.57

# Bugs
## Bug list
- [ ] 1. CLI doesn't work; Only the Python API
- [ ] 1.1. `AttributeError: 'NoneType' object has no attribute 'version'`

## Bug details

### 1. CLI doesn't work; Only the Python API
If you try running via the CLI, using inputs of either (a) OWL, or (b) Obographs JSON, it doesn't work. You'll get errors, for example as seen below.

### 1.1. `AttributeError: 'NoneType' object has no attribute 'version'`
Example command: `python src/oaklib/cli.py --stacktrace -i ~/Desktop/go-nucleus.json dump -o ~/Desktop/go.json -O fhirjson`

Issue is that `source.meta` = `None`. However it's supposed to be an object, w/ a `version` attribute. When I run using the Python API, it works fine. I'm using the `OboGraphToFhirJsonConverter` or `OboGraphToFhirNpmConverter` directly. However, I see that via the CLI, it uses `pronto_implementation.py`, even when I'm passing an (obographs) JSON input. Perhaps it has something to do with that.

Trace

```
python src/oaklib/cli.py --stacktrace -i ~/Desktop/go-nucleus.json dump -o ~/Desktop/go.json -O fhirjson
Traceback (most recent call last):
File "/Users/joeflack4/projects/ontology-access-kit/src/oaklib/cli.py", line 5510, in
main()
File "/Users/joeflack4/Library/Caches/pypoetry/virtualenvs/oaklib-VudrcZnD-py3.9/lib/python3.9/site-packages/click/core.py", line 1130, in __call__
return self.main(*args, **kwargs)
File "/Users/joeflack4/Library/Caches/pypoetry/virtualenvs/oaklib-VudrcZnD-py3.9/lib/python3.9/site-packages/click/core.py", line 1055, in main
rv = self.invoke(ctx)
File "/Users/joeflack4/Library/Caches/pypoetry/virtualenvs/oaklib-VudrcZnD-py3.9/lib/python3.9/site-packages/click/core.py", line 1657, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/Users/joeflack4/Library/Caches/pypoetry/virtualenvs/oaklib-VudrcZnD-py3.9/lib/python3.9/site-packages/click/core.py", line 1404, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/Users/joeflack4/Library/Caches/pypoetry/virtualenvs/oaklib-VudrcZnD-py3.9/lib/python3.9/site-packages/click/core.py", line 760, in invoke
return __callback(*args, **kwargs)
File "/Users/joeflack4/projects/ontology-access-kit/src/oaklib/cli.py", line 2352, in dump
impl.dump(output, syntax=output_type, **kwargs)
File "/Users/joeflack4/projects/ontology-access-kit/src/oaklib/implementations/pronto/pronto_implementation.py", line 590, in dump
super().dump(path, syntax, **kwargs)
File "/Users/joeflack4/projects/ontology-access-kit/src/oaklib/interfaces/dumper_interface.py", line 73, in dump
converter.dump(ogdoc, target=path, **kwargs)
File "/Users/joeflack4/projects/ontology-access-kit/src/oaklib/converters/obo_graph_to_fhir_converter.py", line 100, in dump
cs = self.convert(
File "/Users/joeflack4/projects/ontology-access-kit/src/oaklib/converters/obo_graph_to_fhir_converter.py", line 186, in convert
self._convert_graph(
File "/Users/joeflack4/projects/ontology-access-kit/src/oaklib/converters/obo_graph_to_fhir_converter.py", line 229, in _convert_graph
target.version = source.meta.version
AttributeError: 'NoneType' object has no attribute 'version'
```


# Improvements
## Sub-task list
- [x] 1. Documentation
- [x] 2.1. Edges: simple
- [ ] 2.2. Edges: more
- [ ] 3. Mappings to codes: use CURIEs or CURIE local parts?
- [ ] 4. Ontology metadata
- [ ] 5. Synonym updates
- [ ] 6. Foreign concepts -> External code systems
- [x] 7. Axioms
- [ ] 8. Property/relationship/edge descriptions?
- [ ] 9. ValueSets?
- [ ] 10. Simple CLI
- [ ] 11. Filter
- [ ] 12. Labels -> `concept.property`
- [ ] 13. Non-`is_a` hierarchies
- [ ] 14. Export in FHIR NPM package format
- [ ] 15. SPARQL / DL queries (-> ValueSet, etc)

## Sub-task details
### 1. Documentation
- [x] i. add an exemplar notebook
- [x] ii. add docs to explain end-to-end more clearly, e.g. how first things are loaded into an obograph structure, then what the mapping is from obograph

### 2.1 Edges: simple
- [x] i. Add edges
- [x] ii. Add optionality for more/less predicates (i.e. `--include-all-predicates`)

### 2.2 Edges: more
- [ ] i. Create [ConceptMap](https://build.fhir.org/conceptmap.html) instances? If so, just for things like `skos:exactMatch`, or all edges (except `IS_A`?)?
- [x] ii. Populate `CodeSystem.concept.property`
- [ ] iii. Do we want all edges to be populated into another CodeSystem resource as discussed in the "Foreign concepts -> External code systems" section?

### 3. Mapping to codes: use CURIEs or CURIE local parts?
_Joe_:
a. The consensus seems to be to use `valueCoding` ([`coding`](https://hl7.org/fhir/datatypes.html#Coding)) data type for these. These expect CodeSystem URI, not CURIEs.
b. We could use `valueString` instead and use CURIEs. If so, where to put the `curie_map`/`prefix_map`? Expanding would be impossible without adding additional server functionality. I know that the FHIR RDF group is working on something CURIE related, but I don't know the details.

Also, what to use for the URI? The canonical ontology URI? Or the URI to a copy of the local CodeSystem on the FHIR server (if exists).

Do we want all mapped terms to be populated into another CodeSystem resource as discussed in the "Foreign concepts -> External code systems" section?

### 4. Ontology metadata
Ontology metadata is incomplete.

### 5. Synonym updates
More information in [comments](https://github.com/INCATools/ontology-access-kit/issues/369#issuecomment-1334593723).
- [ ] i. Only exact synonyms, or all? (Joe: I've heard exactSynonyms go in `concept.designation.use`, but other ones go in `concep.property`
- [ ] ii. Use [valueCoding](https://build.fhir.org/datatypes.html#Coding)? If so, how? (I think that's correct and examples are blow)
- [ ] iii. Which synonym types to use?
Chris: Use oio as a code system or map to snomed codes for synonym types?
Joe: Shahim thinks SNOMED is required, but other ones are allowed. So for every instance of a synonym, we can include a copy of each if we want. Example below:
SNOMED (required):
```json
"use": {
"system": "http://snomed.info/sct",
"code": "900000000000013009",
"display": "Synonym (core metadata concept)"
},
"value": "Vertebrata "
```
Ontology's original synonym type (optional):
```json
"use": {
"system": "oio", // <----- needs to be a full URI until FHIR formalizes 'IRI stems'
"code": "hasExactSynonym",
"display": "has exact synonym"
},
"value": "Vertebrata "
```

### 6. Foreign concepts -> External code systems
_Joe_: The FHIR RDF group and others recommended that, if we have any foreign concepts within an ontology (e.g. let's just say that Mondo had a GO class declared somewhere), all of these should be put into a separate CodeSystem resource JSON. I don't know how common this is at all.

Tricky though. It is one thing to want to populate concepts into another CodeSystem resource, but then we also have to have a way to extract other information about that ontology in order to populate the rest of the CodeSystem fields. Could do that by taking the URI to the ontology, downloading it, and then running it through the dumper as well. This would be then a recursive operation, as those ontologies may link to other ontologies.

### 7. Axioms
_Joe_: I haven't looked into this much yet; maybe all of this will be covered while working on _edges_.

### 8. Property/relationship/edge descriptions?
This is the most minor one here. `CodeSystem.concept.property.description` is optional.

Was thinking for example, for RO, could use OLS API to get. Can also include a link to the class's page ([example](https://www.ebi.ac.uk/ols/ontologies/ro/properties?iri=http://purl.obolibrary.org/obo/RO_0002215)).

### 9. ValueSets?
I'm not sure yet if this is something that would be desired, and what kinds of ValueSets might be constructed from a CodeSystem.

### 10. Simple CLI
#### Issues
Problem: We would like to run simple command `runoak -i OWL_PATH dump -o OUTPATH -O fhirjson` and for it to complete quickly. It takes like hours for things like Mondo because of `rdflib`.
Solutions:
a. Need to somehow convert to _Obographs_ under the hood to speed up. I see that this requires some annoying changes in the CLI, since the CLI runs through `main()` before `dump()`, and `main()` will have no idea that it's getting FHIR, so it doesn't know to create an Obograph instead of use `rdflib`; that information only becomes available at the `dump()` phase, after conversion.
b. Pre-convert to sqlite db and use that. Requires: https://github.com/INCATools/ontology-access-kit/issues/405

#### Options to support
i. `include_all_predicates (bool)`: I was thinking a variation on this would be something like `retention` (with options, 'maximal', 'minimal', etc), but this is fine for now.

### 11. Filter
I'm not sure how in demand this would be, but `aehrc/fhir-owl` supports. Can read more in [CodeSystem docs](https://hl7.org/fhir/codesystem.html).

### 12. Labels -> `concept.property`
This is simple and potentially high value. Would enable text search. More info: https://build.fhir.org/codesystem-operation-find-matches.html

### 13. Non-`is_a` hierarchies
There is a need for hierarchies, defined at [`CodeSystem.hierarchyMeaning`](https://hl7.org/fhir/valueset-codesystem-hierarchy-meaning.html), to support other cases, e.g. `part_of`. We could add a flag `--hierarchy-predicate` and defaulting it to `is_a`. FHIR says it supports `is-a`, `part-of`, `grouped-by`, and `classified-with`. Supposedly not extensible.

### 14. Export in FHIR NPM package format
We need to follow what is specified here: https://confluence.hl7.org/display/FHIR/NPM+Package+Specification
An example package: http://hl7.org/fhir/us/core/package.tgz

### 15. SPARQL / DL queries (-> ValueSet, etc)
The main idea here is that we would like to support the ability to extract ValueSets from the CodeSystem. This would probably be in the form of a DL or SPARQL query to select a subset of concepts, but there may be some other use case.

## Related
- User requirements/wishlist: TIMS: (i) https://github.com/HOT-Ecosystem/hapi-fhir-jpaserver-starter/issues/41 , (ii) [google sheet](https://docs.google.com/spreadsheets/d/1oMksaC78pc6ptVvuqifG0_rjMHS4rDffo15qLHmQaGw/edit#gid=0)

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.