INCATools / INCATools/ontology-access-kit

Performance: `SqlImplementation.entity_metadata_map()`

Open
#676 5 comments 1 reaction 0 assignees View on GitHub
Dominant language
Python
Stars
198
Forks
35
Avg merge
3d 4h
Merged PRs (30d)
1

Description

## Overview
This method seems far too slow.

## Example
In [this PR](https://github.com/monarch-initiative/mondo-ingest/pull/363#discussion_r1328031211), using `mondo.db`:

```py
exclusion_rels: List[Tuple[str, str]] = []
for rel in oi.entities_metadata_statements(mondo_ids, ['obo:mondo#excluded_subClassOf']):
if rel:
exclusion_rels.append((rel[0], rel[2]))
```

Each iteration of this loop took on average 15.5 seconds. Estimating that to get through ~25k mondo_ids would take ~100 hours at this rate.

Contributor guide

Open the contributing guide

Research direction

Start with SqlImplementation.entity_metadata_map() and reproduce the example query against mondo.db, measuring the roughly 15.5-second iterations described in the issue. Trace the method's work to identify the source of the slowdown. Done means the example completes substantially faster than the reported estimate for about 25k mondo_ids.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.