OHDSI / OHDSI/WebAPI

init_sqls take too long to execute

Open
#2,451 3 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Java
Stars
151
Forks
183
Avg merge
14m
Merged PRs (30d)
2

Description

When updating our CDM data, I run Achilles and connect its results to ATLAS. However, the Achilles results alone are not sufficient to register a new data source in ATLAS, since ATLAS requires additional tables that Achilles does not create.

So I attempted to create the missing tables using the SQL scripts provided in WebAPI (src/main/resources/ddl/results). After creating the tables, I inserted some initial data using the "init_%" sql files in the same directory.

However, when executing init_concept_hierarchy.sql, the script takes an excessively long time to complete.

From the logs, I found that queries for certain domains are especially slow:
• condition: 7m 30s
• measurement: 8m 59s
• observation: 8m 17s
• procedure: 9m 15s
drug_exposure: 20h 31m 54s (still running)
drug_era: 20h 31m 54s (still running)

In particular, the JOIN operations (e.g., atc5_to_atc3, atc3_to_atc1) appear to be the main bottleneck.

Could you please take a look into this issue?

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the SQL files under src/main/resources/ddl/results, especially init_concept_hierarchy.sql, and reproduce the slow executions for the listed domains. Inspect the atc5_to_atc3 and atc3_to_atc1 joins and their execution times; the work is done when the initialization script completes without the reported excessive delays.

Written by the indexing model from the issue text.

Assessment

Tech stack
sql
Domain
databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.