init_sqls take too long to execute
Nobody has claimed this yet.
- Dominant language
- Java
- Stars
- 151
- Forks
- 183
- Avg merge
- 14m
- Merged PRs (30d)
- 2
Description
When updating our CDM data, I run Achilles and connect its results to ATLAS. However, the Achilles results alone are not sufficient to register a new data source in ATLAS, since ATLAS requires additional tables that Achilles does not create.
So I attempted to create the missing tables using the SQL scripts provided in WebAPI (src/main/resources/ddl/results). After creating the tables, I inserted some initial data using the "init_%" sql files in the same directory.
However, when executing init_concept_hierarchy.sql, the script takes an excessively long time to complete.
From the logs, I found that queries for certain domains are especially slow:
• condition: 7m 30s
• measurement: 8m 59s
• observation: 8m 17s
• procedure: 9m 15s
• drug_exposure: 20h 31m 54s (still running)
• drug_era: 20h 31m 54s (still running)
In particular, the JOIN operations (e.g., atc5_to_atc3, atc3_to_atc1) appear to be the main bottleneck.
Could you please take a look into this issue?
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the SQL files under src/main/resources/ddl/results, especially init_concept_hierarchy.sql, and reproduce the slow executions for the listed domains. Inspect the atc5_to_atc3 and atc3_to_atc1 joins and their execution times; the work is done when the initialization script completes without the reported excessive delays.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- sql
- Domain
- databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100