dice-group / dice-group/gerbil

[QA] Implement unit test for experiment with uploaded files for QA system and dataset

Open
#469 0 comments 0 reactions 1 assignee Claimed by @Hardiksh16 View on GitHub
priority:high type:enhancement
Dominant language
Java
Stars
231
Forks
55
PR merge metrics
No merged PRs in 30d

Description

# Problem

The situation in which a system answer file and a dataset file are uploaded occurs often in practice but there is no unit test covering it. Hence, bugs like the one fixed with commit https://github.com/dice-group/gerbil/commit/794c41a14d73ba554ae28a397d415db0d68ae67b are hard to detect.

# Solution

Implement a unit test that simulates an experiment in which 2 files have been uploaded and are used as prediction and gold standard.
The implementation should be done on the following branch, which still has the bug mentioned above: [QA-name-bug](https://github.com/dice-group/gerbil/tree/QA-name-bug)

The unit test should simulate an experiment task that is created by submitting the following configuration from the UI:
```json
{"type":"QA","matching":"STRONG_ENTITY_MATCH","annotator":[],"dataset":["NIFDS_gold1(gold_1.json)"],"answerFiles":["AF_System(pred_1.json)(undefined)(AFDS_gold_1.json)"],"questionLanguage":"en"}
```
`pred_1.json` and `gold_1.json` should have the same content and may have to be moved into the upload directory as preparation of the unit test. It is expected that the resulting F1-measures are 1.0.

[GoldStdEvalTest](https://github.com/dice-group/gerbil/blob/QA-name-bug/src/test/java/org/aksw/gerbil/execute/GoldStdEvalTest.java) can be used as inspiration.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.