dice-group / dice-group/gerbil

SameAs retrieval causes problems in a JUnit test

Open
#433 0 comments 0 reactions 0 assignees View on GitHub
priority:high type:bug
Dominant language
Java
Stars
231
Forks
55
PR merge metrics
No merged PRs in 30d

Description

## Description

The D2KBTest fails because two entities of the OKE Task 1 Example Dataset seem to be handled in a faulty way.

```
2023-06-05 11:57:52,610 [Thread-7] DEBUG [org.aksw.gerbil.evaluate.impl.ConfidenceBasedFMeasureCalculator] - <${pom_version} 0|||||||||>
2023-06-05 11:57:52,610 [Thread-7] DEBUG [org.aksw.gerbil.matching.impl.MatchingsCounterImpl] -
2023-06-05 11:57:52,610 [Thread-7] DEBUG [org.aksw.gerbil.matching.impl.MatchingsCounterImpl] -
2023-06-05 11:57:52,610 [Thread-7] DEBUG [org.aksw.gerbil.evaluate.impl.ConfidenceBasedFMeasureCalculator] - <${pom_version} 1|||||||||>
2023-06-05 11:57:52,610 [Thread-7] DEBUG [org.aksw.gerbil.matching.impl.MatchingsCounterImpl] -
2023-06-05 11:57:52,610 [Thread-7] DEBUG [org.aksw.gerbil.matching.impl.MatchingsCounterImpl] -
2023-06-05 11:57:52,610 [Thread-7] DEBUG [org.aksw.gerbil.matching.impl.MatchingsCounterImpl] -
2023-06-05 11:57:52,610 [Thread-7] DEBUG [org.aksw.gerbil.matching.impl.MatchingsCounterImpl] -
2023-06-05 11:57:52,610 [Thread-7] DEBUG [org.aksw.gerbil.matching.impl.MatchingsCounterImpl] -
2023-06-05 11:57:52,610 [Thread-7] DEBUG [org.aksw.gerbil.matching.impl.MatchingsCounterImpl] -
2023-06-05 11:57:52,610 [Thread-7] DEBUG [org.aksw.gerbil.matching.impl.MatchingsCounterImpl] -
2023-06-05 11:57:52,610 [Thread-7] DEBUG [org.aksw.gerbil.matching.impl.MatchingsCounterImpl] -
2023-06-05 11:57:52,610 [Thread-7] DEBUG [org.aksw.gerbil.matching.impl.MatchingsCounterImpl] -
2023-06-05 11:57:52,610 [Thread-7] DEBUG [org.aksw.gerbil.matching.impl.MatchingsCounterImpl] -
2023-06-05 11:57:52,610 [Thread-7] DEBUG [org.aksw.gerbil.matching.impl.MatchingsCounterImpl] -
2023-06-05 11:57:52,610 [Thread-7] DEBUG [org.aksw.gerbil.matching.impl.MatchingsCounterImpl] -
2023-06-05 11:57:52,610 [Thread-7] DEBUG [org.aksw.gerbil.matching.impl.MatchingsCounterImpl] -
2023-06-05 11:57:52,610 [Thread-7] DEBUG [org.aksw.gerbil.matching.impl.MatchingsCounterImpl] -
2023-06-05 11:57:52,610 [Thread-7] DEBUG [org.aksw.gerbil.evaluate.impl.ConfidenceBasedFMeasureCalculator] - <${pom_version} 2|||||||||>
2023-06-05 11:57:52,610 [Thread-7] DEBUG [org.aksw.gerbil.matching.impl.MatchingsCounterImpl] -
2023-06-05 11:57:52,610 [Thread-7] DEBUG [org.aksw.gerbil.matching.impl.MatchingsCounterImpl] -
2023-06-05 11:57:52,610 [Thread-7] DEBUG [org.aksw.gerbil.matching.impl.MatchingsCounterImpl] -
2023-06-05 11:57:52,610 [Thread-7] DEBUG [org.aksw.gerbil.matching.impl.MatchingsCounterImpl] -
2023-06-05 11:57:52,610 [Thread-7] DEBUG [org.aksw.gerbil.matching.impl.MatchingsCounterImpl] -
2023-06-05 11:57:52,610 [Thread-7] DEBUG [org.aksw.gerbil.matching.impl.MatchingsCounterImpl] -
2023-06-05 11:57:52,611 [Thread-7] INFO [org.aksw.gerbil.execute.ExperimentTask] -
```

The evaluation brings up the following two mismatches in the second and third document, respectively:
- `((78, 12, [http://dbpedia.org/resource/Bill_Clinton, http://www.ontologydesignpatterns.org/data/oke-challenge/task-1/Bill_Clinton])) != ((78, 12, [http://nl.dbpedia.org/resource/Bill_Clinton, http://nl.wikipedia.org/wiki/Bill_Clinton]))`
- `((44, 6, [http://www.ontologydesignpatterns.org/data/oke-challenge/task-1/Sydney, http://dbpedia.org/resource/Sydney])) != ((44, 6, [http://de.dbpedia.org/resource/Sydney, http://de.wikipedia.org/wiki/Sydney]))`

This raises several questions:
1. Why does the sameAs retrieval provide two different sets of IRIs although it gets the same input documents for the dataset and annotator?
2. Why do we only have language-dependent DBpedia IRIs instead of a larger set of IRIs?

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by running the D2KBTest and tracing the SameAs retrieval for the Bill Clinton and Sydney mismatches. Inspect the logged MatchingsCounterImpl results and the input documents from the OKE Task 1 Example Dataset. Done means explaining why the IRI sets differ and restoring the expected test behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
testing-qa
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.