INCATools / INCATools/ontology-access-kit

Jaccard similarity differences when using information content

Open
#746 2 comments 1 reaction 0 assignees View on GitHub
Dominant language
Python
Stars
198
Forks
35
Avg merge
3d 4h
Merged PRs (30d)
1

Description

I conducted a semantic similarity calculation experiment where I noticed that the Jaccard Similarity Score values of certain records differed when I used the `--information-content-file` option. I am unsure of the reason behind this behavior and have documented the experiment details in case anyone would like to reproduce it. If anyone can explain these differences, I would appreciate it.

The first one was without using any information content files:

```sh
runoak -i semsimian:sqlite:phenio.db similarity -p i \
--set1-file hp_terms.txt \
--set2-file mp_terms.txt \
--min-jaccard-similarity 0.4 \
-O csv \
-o semsim_without_ic_file.tsv

```

Next, I used the same parameters, just including --information-content-file option:

```sh
runoak -i semsimian:sqlite:phenio.db similarity -p i \
--set1-file hp_terms.txt \
--set2-file mp_terms.txt \
--min-jaccard-similarity 0.4 \
--information-content-file phenio_monarch_hp_mp_ic.tsv \
-O csv \
-o semsim_with_ic_file.tsv
```
The HP and MP terms' information content files were generated separately and merged into a final file.

```sh
runoak -i phenio.db -g gene_phenotype.9606.tsv -G hpoa_g2p information-content -p i i^HP: -o phenio_monarch_hp_ic.tsv
```

```sh
runoak -i phenio.db -g gene_phenotype.10090.tsv -G hpoa_g2p information-content -p i i^MP: -o phenio_mp_ic.tsv
```

Here are some exploratory analysis regarding jaccard similarity comparisons

| property | semsim_without_ic | semsim_with_ic |
| :--- | ---: | ---: |
| count | 1,485,387.00 | 1,522,836.00 |
| mean | 0.44 | 0.44 |
| std | 0.03 | 0.03 |
| min | 0.40 | 0.40 |
| 25% | 0.41 | 0.41 |
| 50% | 0.43 | 0.43 |
| 75% | 0.46 | 0.46 |
| max | 0.70 | 0.70 |

Although the percentiles have the same value, 38,798 records differ in their jaccard similarity values. To identify the most extreme differences, I selected the top 10 records. Out of these 10, five showed an increase in the jaccard score value when an external IC file was passed during calculation, and five showed a decrease in the score.

| subject_id | object_id | jaccard_similarity_without_ic | jaccard_similarity_with_ic | difference |
|-----------:|-----------:|------------------------------:|---------------------------:|-----------:|
| HP:0025477 | MP:0013304 | 0.416667 | 0.481481 | 15.56% |
| HP:0025477 | MP:0012070 | 0.416667 | 0.481481 | 15.56% |
| HP:0025477 | MP:0030485 | 0.416667 | 0.481481 | 15.56% |
| HP:0025477 | MP:0031348 | 0.416667 | 0.481481 | 15.56% |
| HP:0025477 | MP:0005422 | 0.416667 | 0.481481 | 15.56% |
| HP:0002514 | MP:0000783 | 0.465116 | 0.425532 | -9.30% |
| HP:0005671 | MP:0000783 | 0.454545 | 0.416667 | -9.09% |
| HP:0007045 | MP:0000783 | 0.454545 | 0.416667 | -9.09% |
| HP:0002514 | MP:0000787 | 0.5 | 0.458333 | -9.09% |
| HP:0005849 | MP:0000783 | 0.454545 | 0.416667 | -9.09% |

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the two `runoak similarity` commands using `hp_terms.txt`, `mp_terms.txt`, and the merged information-content file. Compare how the `--information-content-file` option affects Jaccard calculations, then determine whether the differing records are expected or indicate a bug; the issue is done when this behavior is explained and any needed correction is identified.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, sqlite
Domain
cli, data
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.