INCATools / INCATools/semantic-sql
Improve speed of converting SQLite to FHIR
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 69
- Forks
- 7
- Avg merge
- 8m
- Merged PRs (30d)
- 1
Description
Overview
I tried to convert HPO to FHIR using semsql as an intermediary. However, after about 40 minutes, I decided to give up and switch to Obographs for speed. I think it took about 10 minutes to convert to a .db, and the rest of the time in my process was just OAK trying to load the DB. Normally semsql is much faster to load than using rdflib, but not in this case. I looked and saw that my hpo.db was about 1GB, which is about 10x larger than my hpo.owl. I looked at some of my other conversions, and it looks like this 5-10x file size was normal.
If I'm correct that the issue is not so much OAK performance, but just the file size in general, is there anything we can do to reduce these file sizes? Or maybe it's not so much the size, but the structure that is taking OAK a long time to parse downstream? If this is more of an OAK issue (or both an OAK issue and a semsql issue), I can open up a ticket over there.
Potential causes
May be 1 or more of the following that's taking a lot of time.
a. Semsql: File size
b. Semsql: Non-optimal structures for downstream parsing
c. OAK: Not parsing optimally
d. OAK: Spending time doing things that are maybe not needed for my use case
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by examining the semsql conversion path and the resulting hpo.db versus hpo.owl sizes described in the report. Profile both SQLite creation and downstream OAK loading to determine whether file size, database structure, or unnecessary parsing work causes the delay; done means identifying the responsible component and a concrete optimization path.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, sqlite
- Domain
- databases, performance
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100