dice-group / dice-group/gerbil

Fine grained (per annotation) experiment results

Open
#331 0 comments 0 reactions 0 assignees View on GitHub
type:enhancement
Dominant language
Java
Stars
231
Forks
55
PR merge metrics
No merged PRs in 30d

Description

### User Story

As the developer of an annotation system, I would like to understand the results of GERBIL in detail to be able to enhance my annotation system. This goes beyond the summaries GERBIL is showing in the UI. Instead, I would like to see exactly which annotation has been marked as tp, fp or fn.

### Current situation

When changing the level of the logging to `DEBUG` (can be done in the [`log4j.properties` file](https://github.com/dice-group/gerbil/blob/master/src/main/properties/log4j.properties#L2) by replacing the keyword `WARN`) the class which is counting the tp, fp and fns will log them.
```
2019-09-12 12:46:41,816 [pool-1-thread-1] DEBUG [org.aksw.gerbil.evaluate.impl.ConfidenceBasedFMeasureCalculator] - <${pom_version} 0|||||||||>
2019-09-12 12:46:41,867 [pool-1-thread-1] DEBUG [org.aksw.gerbil.matching.impl.MatchingsCounterImpl] -
2019-09-12 12:46:41,868 [pool-1-thread-1] DEBUG [org.aksw.gerbil.matching.impl.MatchingsCounterImpl] -
2019-09-12 12:46:41,883 [pool-1-thread-1] DEBUG [org.aksw.gerbil.matching.impl.MatchingsCounterImpl] -
2019-09-12 12:46:41,883 [pool-1-thread-1] DEBUG [org.aksw.gerbil.matching.impl.MatchingsCounterImpl] -
2019-09-12 12:46:41,885 [pool-1-thread-1] DEBUG [org.aksw.gerbil.evaluate.impl.ConfidenceBasedFMeasureCalculator] - <${pom_version} 1|||||||||>
2019-09-12 12:46:41,886 [pool-1-thread-1] DEBUG [org.aksw.gerbil.matching.impl.MatchingsCounterImpl] -
2019-09-12 12:46:41,886 [pool-1-thread-1] DEBUG [org.aksw.gerbil.matching.impl.MatchingsCounterImpl] -
2019-09-12 12:46:41,886 [pool-1-thread-1] DEBUG [org.aksw.gerbil.matching.impl.MatchingsCounterImpl] -
...
```
The `ConfidenceBasedFMeasureCalculator` class prints delimiter lines to separate the single documents (nobody knows why the line contains things like `${pom_version}`... but never touch a running system :wink: ). The `MatchingsCounterImpl` class prints the tp, fp and fn annotations found.

However, this has several disadvantages:
* The user has to go through the logs :roll_eyes:
* Depending on the experiment type, GERBIL will run several evaluations for the different classes of entities (emerging entitites, etc.). This leads to the situation that the evaluation has to be executed several times with different sets of entities which leads to printing the log messages of the single documents several times with different content (depending on the entity classes).

Overall, this is horrible to work with :disappointed:

### First aid :face_with_head_bandage:

It would already help if the `ClassConsideringFMeasureCalculator` class could log which evaluation it executes ("default" evaluation in [line 62](https://github.com/dice-group/gerbil/blob/master/src/main/java/org/aksw/gerbil/evaluate/impl/ClassConsideringFMeasureCalculator.java#L62), class-based evaluation in [line 69](https://github.com/dice-group/gerbil/blob/master/src/main/java/org/aksw/gerbil/evaluate/impl/ClassConsideringFMeasureCalculator.java#L69)). This would at least make the logging more readable and usable.

### Developer enhancement

As a developer, I would like to have the documents (e.g., to a file) including gold standard annotations and annotations of my system. The annotations should be marked with tp, fp and fn. In case of tp, the matching annotation should be linked.

The following (coarse grained) steps woudl be necessary:
- [ ] Enhance the interfaces of the evaluation classes (mentioned above) by letting them assign the single annotations to lists (or sets) of tp, fn and fp annotations. Each document would need to have its own list leading to a list of these lists when it comes to the evaluation of the dataset.
- [ ] Each annotation would have several "classes":
- The source of the annotation (the gold standard or annotation system)
- The evaluation result (i.e., tp, fn and fp)
- Additional classes if available (e.g., emerging entity)
- [ ] Since GERBIL already has a NIF file generating mechanism, reusing it to write these detailed results to a file seems to be the easiest approach.
- [ ] It is important to be able to configure the generation of such a file using the `gerbil.properties` file.

### Far away, shiny future :sparkles:

For publicly available datasets, the UI should offer the visualisation of these detailed results.

- [ ] The dataset's metadata need to carry the information whether it is allowed to show these results for the dataset. We have to exclude several datasets that are not publicly available.
- [ ] The UI has to be extended to offer an additional view with which the user can dive into these details.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.